Deep Confident Steps to New Pockets: Strategies for Docking Generalization
Authors: Gabriele Corso
Arthur Deng
Nicholas Polizzi
Regina Barzilay
Tommi S. Jaakkola
Venue: ICLR 2024 poster (2024)
Keywords: generalization
molecular docking
protein-ligand binding
diffusion models
benchmark
bootstrapping
self-training
Primary area: applications to physical sciences (physics, chemistry, biology, etc.)
TL;DR: A new benchmark to test docking generalization and a novel self-training technique to learn without access to data

Abstract: Accurate blind docking has the potential to lead to new biological breakthroughs, but for this promise to be realized, docking methods must generalize well across the proteome. Existing benchmarks, however, fail to rigorously assess generalizability. Therefore, we develop DockGen, a new benchmark based on the ligand-binding domains of proteins, and we show that existing machine learning-based docking models have very weak generalization abilities. We carefully analyze the scaling laws of ML-based docking and show that, by scaling data and model size, as well as integrating synthetic data strategies, we are able to significantly increase the generalization capacity and set new state-of-the-art performance across benchmarks.  Further, we propose Confidence Bootstrapping, a new training paradigm that solely relies on the interaction between diffusion and confidence models and exploits the multi-resolution generation process of diffusion models. We demonstrate that Confidence Bootstrapping significantly improves the ability of ML-based docking methods to dock to unseen protein classes, edging closer to accurate and generalizable blind docking methods.

Published as a conference paper at ICLR 2024
DEEP CONFIDENT STEPS TO NEW POCKETS: STRATEGIES FOR DOCKING GENERALIZATION
Gabriele Corso" Arthur Deng"?, Nicholas Polizzi", Regina Barzilay', Tommi Jaakkola'
"SSAIL, Massachusetts Institute of Technology. Universityy of California, Berkeley, 3 Darnarrarber Cancer Institute and Harvard Medical School
ABSTRACT
Accurate blind docking has the potential to lead to new biological breakthroughs, but for this promise to be realized, docking methods must generalize well across the proteome. Existing benchmarks, however, fail to rigorously assess generaliz- ability. Therefore, we develop DOCKGEN, new benchmark based on the ligand- binding domains of proteins, and we show that existing machine learning- based docking models have very weak generalization abilities. We carefully analyze the scaling laws of ML-based docking and show that, by scaling data and model size, as well as integrating synthetic data strategies, we are able to significantly in- crease the generalization capacity and set state-of-the art performance across benchmarks. Further, we propose CONFIDENCE BOOTSTR APPING, a new train- ing paradigm that solely relies on the interaction between diffusion and confidence models and exploits the multi-resolution generation process of diffusion models. We demonstrate that CONFIDENCE BOOTSTRAPPING significantly improves the ability of ML-based docking methods to dock to unseen protein classes, edging closer to accurate and generalizable blind docking methods.
INTRODUCTION
Understanding how small molecules and proteins interact, a task known as molecular docking, is at the heart of drug discovery. The conventional use of docking in the industry has led the field to focus on finding binding conformations when restricting the search to predefined pockets and eval- uating these on relatively limited set of protein families of commercial interest. However, solving the general blind docking task (i.e. without pocket knowledge) would have profound biological implications. For example, it would help us understand the mechanism of action of new drugs to accelerate their development [Schottlender al., 2022], predict adverse side-effects of drugs be- fore clinical trials [Luo et al., 2018], and discover the function of the vast number of enzymes and membrane proteins whose biology do not yet know [Yi et al., 2015]. All these tasks critically require the docking methods to generalize beyond the relatively small class of well-studied proteins for which we have many available structures.
Existing docking benchmarks are largely built on collections of similar binding modes and fail to rig- orously assess the ability of docking methods to generalize across the proteome. Gathering diverse data for protein-ligand interactions is challenging because binding pockets tend be evolutionarily well-conserved due to their critical biological functions. Therefore, a large proportion of known interactions fall into relatively small set of common binding modes. Moreover, human biases in the collection of binding conformational data further compromise the representativeness of existing benchmarks. To address these problems, we propose DOCKGEN, new benchmark that aims to test a method's 's ability to generalize across protein domains. With DOCK GEN, we show that existing machine learning-base based docking methods poorly predict binding poses on unseen binding pockets.
With this new benchmark, we analyze the scaling laws of DIFFDOCK, the state-of-the-art MI-- based blind docking method, with respect to the size of both its architecture and its training data. The results show that increasing both data and model can give significant generalization improve- ments. Further, we devised and integrated a synthetic data generation strategy based on extracting side chains as ligands from real protein structures. Putting these together, our new DIFFDoCK-L.
"Equal contribution. Correspondence gcorso@mi ededu

---

Published as a conference paper at ICLR 2024
(0) high
confidence
Pe (x x(1)))
diffusion generation rollouts
(0)
(x)))x10)
confidence
(t)
Bootstrapping finetuning
Figure Visual representation of the CONFIDENCE BOOTSTRAPPING training scheme. The dashed lines represent the reverse diffusion generation rollouts that the model executes. The dotted lines illustrate the bootstrapping feedback from the confidence model that is used update the like- lihood of the early diffusion steps by changing the weights of the score model. The pink regions of the protein represent areas to where the docking algorithm is still attending, which starts from being the whole protein and then gradually narrows to the local environment around the current pose.
increases the top ML-based DOCKGEN performance from 7.1 % to 22.6% setting new state-of-the art. However, with the current data and computing resources available today, this trend alone might not be sufficient to fully bridge this generalization gap.
To move beyond this challenge, we propose CONFIDENCE BOOTSTRAPPING, a novel self-training scheme inspired by Monte Carlo tree-search methods, where fine-tune directly protein-ligand complexes from unseen domains without access to their structural data. The fine-tuning is enabled by the interaction between diffusion model rolling out the sampling process and confidence model assigning confidence scores the final sampled poses. These confidence scores are then fed back into the early steps of the generation process (see Figure I for visual representation), This process iterated to improve the diffusion model's performance unseen targets, effectively closing the generalization gap between the diffusion model and the confidence model.
We test CONFIDENCE BOOTSTRAPPING the new DOCKGEN benchmark by fine-tuning small and efficient version of DIFFDOCK on individual protein clusters. each of these clusters, within the first few iterations of bootstrapping, the diffusion model is pushed to generate docked poses with increasingly high confidence. This increased confidence also translates into significantly higher accuracy with the fine-tuned models improving from 9.8% to 24.0% success rate overall, and above 30% in half of the protein domains.
2 RELATED WORK
Search-base based docking Due to its importance in biological research and drug discovery, molecular docking has for decades been central challenge for the computational science community [Halgren et al., 2004; Jain, 2003; Thomsen & Christensen, 2006]. Originally, most techniques followed the search-based paradigm, which is still prevalent today. These methods consist of scoring function and an optimization algorithm. The latter searches over thousands or millions of different confor- mations, which are passed the scoring function that determines their iikelihood/ooddnes.. While these methods tend show relatively robust generalization across protein classes, they are signifi- cantly affected by the size of the search space, which grows exponentially as the ligand gets larger or as assumptions are removed (e.g. receptor rigidity).
ML.-based docking Researchers have recently tried to move beyond the search-based paradigm and directly generate poses with deep learning models. The first attempts [Stärk et al., 2022; Lu al., 2022] framed the docking task as regression problem; this showed significant improvements in runtime but did not reach the accuracy of search-based methods. Corso et al. [2022] proposed DIFFDOCK, generative model based on the diffusion modeling framework that is trained sample
2

---

Published as a conference paper at ICLR 2024
A.
Potsoel Natowet
DeckCte Defoe!
230
11
0.
02 F7
F3
A. An example of the superimposition of the pockets of two proteins PDBBind, IQXZ in pink and 5M4Q in cyan, that share very similar binding pocket structure (a bound ligand shown in red), but have only 22% sequence similarity. While sequence similarity splits would classify them in separate clusters, our approach correctly identifies that the binding domain of these two proteins is the same. B. Comparison of binding sites in train test set for both PDBbind and DOCKGEN datasets. BLOSUM62 and harmonic mean similarity metrics (more details in Appendix A.2) have a maximum of (most similar) and minimum of 0 (least similar). The densities clipped at 1% of the maximum value for both datasets to emphasize contamination. Every binding site in the train set compared to every binding site in the test set showing significantly higher train-test similarity in the PDBBind dataset compared to the DOCKGEN dataset.
Figure 2:
docked ligand poses. In particular, DIFFDOCK diffusion model to sample small number of possible poses that are then passed to a confidence model that ranks and assigns a score to each.
Blind docking benchmarks The majority of previous ML-based methods used the PDBBind dataset [Liu et al., 2017], a curated set of protein-ligand crystallographic structures from PDB [Berman et al., 2003], to train and test models. In particular, they adopted time-based split of the dataset where structures that were resolved before 2019 went into training and validation, and those from 2019 formed the test set. Stärk et al. [2022] and others also evaluate the performance on reduced set of proteins with different UniProt-ID [Consortium, 2015] compared to those in the train- ing set. Here, while ML methods show drop in performance, they remain in line with search-based techniques [Corso al., 2022]. Similarly, concurrent works [Masters et al., 2023; Buttenschoen et al., 2024] define new splits or benchmarks based on global sequence similarity.
THE DOCKGEN BENCHMARK
We argue that the existing approaches used to evaluate the ML docking methods fall short in ana- lyzing their generalization capacity to different parts of the proteome. Binding pockets, due their importance to many biological mechanisms, are often among the most well-conserved regions in proteins. Therefore, just looking at the UniProt-ID of a protein its global sequence similarity often leads to proteins in the train and test sets that have the same underlying pocket. Figure 2-A shows an example of such failures, where two proteins, (30% is often used as cutoff), share very similar binding pockets.
with only 22% sequence similarity
To better detect these cases we delve one level deeper into the organization of proteins and look at protein domains. Protein domains are the structural and functional units that compose proteins. Very similar domains can appear in different sequences but have similar local structural characteristics. Therefore, by looking at the protein domains where ligands bind, we can form a more granular classification of the protein-ligand interactions.
To classify each domain, we used the ECOD [Cheng et al., 2014] classification. This clustering divides the 17k complexes from PDBBind before 2019, which have been used for training and validation of previous ML models, into 487 clusters. The remaining data from 2019, from which the test set was generated, presents only additional clusters composed of a total of 15 complexes. This clustering approach is very different from that taken by other methods based on global sequence similarity.
3

---

Published as a conference paper at ICLR 2024
Table 1: Top-1 RMSD performance of different methods the PDBBind and DOCKGEN bench-
marks. Runtimes were computed as averages over the PDBBind test set. run on CPU. Med. indicates the median RMSD. Ex. refers the level of exhaustiveness of the search in case this was increased above the default value. 'more details in Section 5.
PDBBind Med.
DocKGEN-full Med.
DockGEN-cluterss Med.
Average Runtime (s)
Method
%<2Å
%<2A
%<<Å
SMINA
18.7
7.1
7.9
13.8
2.4
16.4
126
SMINA (EX. 64)
25.4
5.5
10.6
13.5
4.7
14.7
347
P2RANK+SMINA
20.4
4.3
7.9
14.1
16.4
126
GNINA
22.9
7.7
14.3
15.2
9.4
14.5
127
GNINA (EX. 64)
32.1
4.2
17.5
8.1
11,8
6.2
348
P2RANK+GNINA
28.8
4.9
13.8
16.2
4.7
15.3
127
EQUIBIND
5.5
6.2
0.0
13.3
13.3
0.04
TANKBIND
20.4
4.0
0.5
11.6
0.0
11.1
0.7
DIFFDOCK (10)
35.0
3.6
7.1
6.8
6.1
6.0
10
DIFEDOCK (40)
38.2
3.3
6.0
7.3
3.7
6.7
40
DIFFDOCK-L (10)
43.0
2.8
22.6
4.3
27.6
3.7
25
DIFFDOCK-S C.B. (10)
24.0
3.8
2.8
To obtain a more sizable test set without retraining the models on a reduced set, we turn to the Bind- ing MOAD dataset [Hu et al., 2005]. Similar to PDBBind, Binding MOAD is curated collection of protein-ligand complexes from the PDB. However, due its different filtering and requirements (e.g. no requirement for known affinity), it contains a set of 41k complexes partially disjoint from PDBBind. These come from 525 ECOD clusters, 346 of which in common with PDBBind, and 179 of which are not present in PDBBind.
To generate the validation and test datasets of the new benchmark, we randomly divide these remain- ing clusters in two and then apply number further filtering steps (more details in Appendix A). In particular, we exclude protein-lige ligand complexes with multiple ligands interacting in the same pocket (i.e. no additional bound cofactors). We also remove metals, crystal additives, and large molecules with more than 60 heavy atoms. To maintain a chemical balance we only keep up to 5 complexes with the same ligand in the validation and test datasets. This leaves us with 141 complexes in the validation and 189 complexes in the test set. A careful analysis the binding-site similarity of the different datasets highlights the vast improvement brought by DOCKGEN in terms of binding site generalization (Figure 2.B and Appendix A.2).
We then run number of baselines considered to be the state-of of-the-art open-source or open-access models: for search- based methods, SMINA [Koes et al., 2013] and GNINA [McNutt al., 2021], while for ML methods, EQUIBIND [Stärk et al., 2022], TANKBIND [Lu et al., 2022] and DIFFDOCK [Corso al., 2022]. Since search-based methods have been shown to improve their blind docking performance by first selecting a pocket with a pocket finder method like P2RANK [Krivák & Hoksza, 2018], also report these performances.
Previous ML methods significantly underperform in this new benchmark (Table 1), and their perfor- mances are only fraction of those that they have in the time-split complexes from the PDBBind test set, with regression methods having nearly no success. On the other hand, search-based methods have a significantly lower drop in performance, but even when increasing the exhaustiveness of their search, they not able to find the true pose in the vast majority of cases, motivating the need for the development of more accurate methods.
CONFIDENCE BOOTSTRAPPING
Docking, along with other structural biology problems like protein folding, has been commonly treated as NP-hard combinatorial optimization problem [Thomsen, 2007; Mukhopadhyay, 2014], Although the success ML methods such as AlphaFold2 [Jumper et al., 2021] has demonstrated that deep learning methods can significantly shorten the search in practice. The NP perspective suggests a useful insight into the problem: is easier to check that a pose is good than generate a good pose. This perspective points towards the exploration of new self-training-bar where feedback from discriminative model is used to update the generative model exploring the
strategies
4

---

Published as a conference paper at ICLR 2024
space of conformations and help the latter generalize to protein domains where we have no data for ground truth poses.
4.1 BACKGROUND
Diffusion models Let p(x) be some data distribution of interest. Score-based diffusion gener- ative models [Song et al., 2021] are trained to sample from p by defining a continuous diffu- sion process dx f(x, t)dt + g(t)dw, where represents the Wiener process that transforms the data distribution in a simple prior, and learns to reverse such a process. This is enabled by the existence of corresponding reverse diffusion process which can be expressed by the SDE dx = [](x,)) g())² Vx log Pe (x) + g())dw where pr(x) is the likelihood of the evolving data distribution. To run this reverse diffusion equation, we need to learn to approximate the score of the evolving data distribution (x t) x Vx (x). This achieved by optimizing the parameters via the denoising score matching loss:
8" = arg min [EL-UO.TT] { ((t)) pxto pxto) [1150(x(1), - V Vxco log poe (x(()) |x (0)
where U refers the uniform distribution, X(t) is a positive weighting function and Po: the transition kernel of the forward diffusion process. One view of diffusion models is via the lens of their iterative denoising generation ladder which, at every step, exponentially reduces the size of candidate poses' posterior distribution (progressively higher resolution), a perspective we will use to motivate our approach. Diffusion models were also generalized to compact Riemannian manifolds [De Bortoli al., 2022]. a formulation that particularly well suited for scientific applications where the main degrees of freedom can be well represented by actions on a low-dimensional Riemannian manifold [Corso, 2023]. This idea underlies DIFFDOCK and other recent advances in computational sciences [Jing et al., 2022; Watson et al., 2023].
Self-training methods Self-training refers to range of techniques that have been employed in several different ML application domains where labels predicted by some model unlabelled data are used for training. For example, in the setting of image classification, Xie et al. [2020] used unlabelled images to improve a classifier by first generating labels from the clean image with the current classifier version, and then training the classifier to make the same prediction on noisy ver- sions of the same image. This method was taken as inspiration for the self-distillation technique used by AlphaFold2 [Jumper et al., 2021], where after first training iteration, predicted structures satisfying a certain residue pairwise distance distribution were used for a second iteration of model training.
In the realm of generative models, McClosky et al. [2006] used the labels predicted by discrimina- tive reranker to select the best parses generated by a generative parser and add them to the training set. Jin et al. [2021] took similar approach for antibody optimization via the feedback of a neutral- ization predictor. Finally, Generative Adversarial Networks (GANs) [Goodfellow et al., 2014] also use the feedback from discriminator to train a generative model. However, in GANs one relies on having in-distribution data to jointly train the discriminator.
4.2 METHOD
These existing self-training methods do not, however, directly exploit the structure of the generative process they optimize. Moreover, they often fail if the initial generator has a low signal-to-noise ratio. Iterating such generators amplifies the initial errors [McClosky al., 2006]. We argue that diffusion models, because of their multi-resolution structure, offer a unique opportunity to more precisely target the effect of self self-training and avoid error amplification.
large challenge for the diffusion model is that in the first steps of the reverse diffusion process, the model needs to determine both the pocket and approximate pose of the ligand, without having clear view of how well that pose will fit in the pocket. If the model finds that the pocket does not fit the ligand adequately after the ligand has been partially docked, the model can not backtrack its decisions' or learn from its mistakes.
NNoe that in theory at every step the conditional distribution learned by the diffusion model spans the whole search space; however, practice the model learns, based on the prior, to not to look for conformation beyond distance that much larger than the current noise level.
5

---

Published as a conference paper at ICLR 2024
We introduce CONFIDENCE BOOTSTRAPPING, a training mechanism that refines diffusion gen- erator based on feedback from a confidence model. The diffusion model used to "roll out" the reverse diffusion process, generating poses that are then scored with the confidence model. These scores are used inform how to update the parameters of the early steps of the diffusion process so that the model will generate more poses close to those with high confidence (see graphical representation in Figure 1). This process is then repeated for several steps.
There are several reasons why we hypothesize that the diffusion model would be able to learn from the feedback of the confidence model. As discussed above, while generating a ligand pose for new protein is difficult, testing whether a pose is satisfactory is a local and simpler task that, we believe, can more easily generalize to unseen targets. Intuitively, the more difficult the underlying task the more a model is likely to overfit, capturing some spurious correlations in the data instead of the underlying signal. Similarly, early steps of the reverse diffusion process that have to decide the most likely among the different pockets and poses will struggle more than the late steps where the remaining posterior is significantly more localized. Training the early steps of the reverse dif- fusion based on the confidence outcomes of the rollouts, CONFIDENCE BOOTSTRAPPING is able to exploit the multi-resolution structure of diffusion models and close the generalization gap between the different diffusion steps.
From this perspective, our procedure of (1) rolling out the steps of the generation process, (2) eval- uating the success at the end, and (3) feeding back information from the end to the initial steps resembles the Reinforcement Learning (RI.) algorithm used to master games such as Go [Silver et al., 2016]. Instead of score and confidence models, they use policy and value networks, and the diffusion process is replaced by Monte Carlo tree-search. More generally, our problem can be loosely seen as an RI. problem, where confidence is the reward. We discuss this connection in more detail in Appendix D.
Although it adds complexity to the training process, CONFIDENCE BOOTSTRAPPING is particularly well suited for the molecular docking task. Firstly, as discussed in Section 3, the limited amount of training data and its lack of diversity make alternative training methods critical for the success of blind docking. Furthermore, CONFIDENCE BOOTSTRAPPING can leverage information from very large affinity datasets such as BindingDB [Liu et al., 2007]. Exploiting this binding data is largely unexplored for the docking task. Finally, docking screens are usually run on very large number of complexes (up to tens of billions [Gorgulla et al., 2020]) using restricted set of proteins. Therefore, any time that one would spend fine-tuning the docking algorithm for the restricted set would be largely amortized throughout the screen.
4.3 FORMALIZATION
We now formalize the CONFIDENCE BOOTSTRAPPING training routine. For simplicity, we will present it based on the diffusion model formulation in Euclidean space. However, note that this di- rectly applies to the Riemannian diffusion formulation [De Bortoli et al., 2022] used by DIFFDOCK.
Let po (x; d) be the probability distribution of poses learned by the diffusion model with score so so(x(t), d)) where d is the known information about the complex (e.g, protein and molecule iden- tity). Let (x. d) be the output of the confidence model, and let D = {d1, d2,.... be set of known binders (e.g. from BindingDB) for the target distribution of interest.
CONFIDENCE BOOTSTRAPPING consists of K iterations where at each iteration i the score model weights 0 are updated based as following optimization:
gi+1 arg min EE-UO { x(t) Ex dd-paan Ex peto: [1100(e(0) (t,)))) - V Vx(o log Poe (x x (0))
0
+ X() Beso dr is (x [xxa(x(0) (1;d) V log (x || }
where 80 are the weights of the pretrained diffusion model (if not trained from scratch), and Pa.6 (x, d) x po(x;d) d) explc, d)..
Each of these iterations i [0, K) is achieved by performing rollout stage, followed by an update stage. During the rollout stage, we first sample d from D, then sample points from po: (-.d). forming a buffer B = [x1 .... During the update stage, a fixed number of stochastic gradient descent steps are performed where half of the fine-tuning complexes are taken from the training dataset (first

---

Published as a conference paper at ICLR 2024
half of the objective) and half are taken from the buffer B (second half). In particular, to approximate samples from por d), the elements (x, d) of B are sampled with probabilities proportional to exples(x; d)].
Using different À and X' allows us to take advantage of the multiresolution decomposition of dif- fusion models and direct the bootstrapping feedback principally to update the initial steps of the reverse diffusion. The samples taken from the combination of diffusion and confidence models are likely to be too noisy to provide fine-grained guidance for small t. To prevent the model from for- getting how to perform the final steps, we still take samples from the training dataset and give them a larger weight for small t. Further details on the implementation and optimization of this routine can be found in Appendix C.
EXPERIMENTS
5.1 ANALYZING DOCKING SCALING LAWS
Empirically, we first analyze the effect that scaling the training data and the model size has on the generalization capacity of DIFKDOCK. This analysis is important to evaluate the potential impact of community efforts to increase the amount of available high-quality training data and develop large-scale MI. docking models.
Increasing the training data We investigate how much the addition of more data, within the pro- tein domains that the model already sees, can help with generalization outside of those domains. This is an important question because in recent years there has been call from the research com- munity for pharmaccutical companies to release the large number of crystallography structures they possess. We test this question by including in the training data all MOAD complexes from the same binding protein domains as those seen in PDBBind training and validation sets and released before 2019 (to maintain the validity of the benchmarks). After filtering out duplicates and non con-conformant ligands, this increases the number of training data points by approximately 52%.
Van der Mer-inspired docking augmentation Additional training data from MOAD contributes modestly to pocket diversity, as the extra data points lie within the same protein domain clusters. To increase pocket diversity, we design a novel auxiliary training task based on the generation of synthetic docked poses using protein sidechains as surrogate ligands. We take inspiration from the concept of a van der Mer (vdM). which has been used successfully in the design of proteins that bind target ligands [Polizzi & DeGrado, 2020]. A van der Mer is an amino acid that interacts with another amino acid that is distant in the ID protein sequence, which can closely approximate a noncovalent protein-ligand interaction. In given protein crystal structure, number of protein contacts distant in sequence. The interacting amino acids are assigned as the "binding pocket" for the chosen sidechain. We remove the coordinates of the "ligand" residue and its sequence- local neighbors from the protein to generate the new target complex (more details on how exactly these are chosen can be found in Appendix B.2).
select sidechain with large
The advantage of these synthetic protein-ligand complexes is that they are myriad and easy com- pute, since any (even unliganded) protein structure can be used to generate such examples. Thus, potentially dramatically increase the structural and chemical diversity of binding domains and pockets. This diversity could help the model understand the chemical and geometrical envi- ronments of different pockets. The drawbacks are that these synthetic complexes are of unknown affinity (many could be weak binders), and the chemical diversity of ligands is limited to the 20 amino acids.
Increasing the model size Further, we evaluate how the generalization ability of the docking models changes when scaling their capacity. The relationship between model capacity and gener- alization has been a topic of significant study in the machine learning community. On one hand, the traditional learning theory perspective suggests that once the model overfits to some amount of training data increasing the model size will only likely make the overfitting worse. On the other hand, recent evidence for deep learning models has shown the opposite behavior with "overparame- terized" models increasing their ability to generalize when given more capacity/ [Belkin al., 2019]. We compare score models of the size of the original DIFFDOCK (~~0MM parameters) with others of smaller (~4M) and larger ones (~30M).
7

---

Published as a conference paper at ICLR 2024
25
POBBind
DiffDock-L
POBBInd BIndingMOAD
POBBind BindingMOAD + vdM
20
PDBBind BindingMOAD + vdM inference tuning
RMSD <
15
10
SMINA
GNINA
TANKBind
DifiDock
4M 20M par.
30M par:
Baseline methods
original
Scaling DifiDock
Figure 3: Analysis of the scaling laws of DifiDock when measuring its ability generalize unseen protein domains. par indicates the number of parameters and the different colors indicate different training sets and augmentations. For the 30M architecture, only model was trained due to its expensive training cost. Inference tuning refers to tuning various reverse diffusion hyperparameters at inference time.
Experimental results The results of these experiments, shown in Figure 3, highlight a clear trend of improvement both when increasing the training data and when increasing the size of the model. The vdM augmentation strategy also seems to provide some improvements when scaling to larger model sizes. Combining these components we develop DIFFDoCK-L, which we release publicly². DIFFDOCK-L improves the ML-docking performance on DOCKGEN from 7.1% to 22.6%, even outperforming the best search-basee method (17.5%). DIFFDOCK-L also achieves state-of-the-an both PDBBind (see Table 1) and PoseBusters test (see Appendix E. I). Overall we believe that the analysis of these trends may prove useful for many practitioners and support the effort of the community in developing even larger open-source datasets and models.
blind docking performance
5.2 CONFIDENCE BOOTSTRAPPING
the new DOCKGEN benchmark, where we fine-tune a model on each protein domain cluster. In particular, we use DIFFDOCK-S, the 4M parameters version of DIFFDOCK introduced in Appendix C.2. For computational feasibility, we use clusters with at least 6 complexes and restrict the test set separate clusters (5 for validation) for a total of 85 complexes, which compose the DOCKGEN-clusters subset.
We test CONFIDENCE BOOTSTRAPPING
As can be from Figure 4-A, for most of the clusters. the median confidence of the predicted structures increases along the fine-tuning. Critically, Figure 4-B and 4-C show that also the accuracy of most clusters significantly improves over the course of the bootstrapping process. In Figure 4-D, we plot the average performance across all clusters in comparison to that of the baselines. From this, we see that, in GEN-luttssssserrss CONFIDENCE BOOTSTRAPPING considerably raises the baseline DIFrDocK-S' performance going from 9.8% to 24.0% and doubles that of the traditional bearch-based methods even when run with high exhaustiveness.
The analysis becomes in the individual clusters. In half of the clusters, the model is able to reach top-1 RMSD < 2 performance above 30%. These clusters mostly constitute those which the original model has non-zero accuracy with an initial performance varying from around 2% to 20%. Then we have one cluster where the accuracy is improved to only ~00% and three clusters where the model never selects good poses neither before nor after the bootstrapping. These results, further supported by the performance when using oracle confidence model in Appendix E.2, suggest that future
more interesting when looking into the evolution of the performance
We release data, instructions, code, and weights at https: httt::///gthuut com/ http://ww..com/gccro//////D// Coode for these experiments be found https:/ www.ttt.....m.eeeeeeeeeeeeeeeeee wwt../cem//.////////..e. wetiiennnebbotttrreee
8

---

Published as a conference paper at ICLR 2024
100
10
fluorinating dnayme...
BDDbinddin domain
Median Confidence Score
Carbon-nitrogen hydrolese
Nolysdenum biesynthesis...
6
Homo-algomen algomeric flvvincontaining...
Nuclevside binding domain
4
SGNH hydrolase
Gutaminste/kpereginnnee.
RMSD
40
0
20
-4
-6
0
10
20
30
40
50
60
10
20
30
40
50
60
A.
Bootstrapping iterations
Bootstrapping Iterations
Bootstrapping
After Bootstrapping
80
RMSD 2A (%)
60
RMSD <
20
40
15
Fine-lured Diff k-5
10
DimDock
20
5
SHINA (ex. 64)
DiftDock
2
3
5
6
10 20 30
40
50
60
C.
Test Clusters
Bootstrapping Iterations
Figure 4: Empirical performance of CONFIDENCE BOOTSTRAPPING across the 8 protein domain clusters within DoCK GFN-cluster. We did two fine-tuning runs each cluster and report the averaged results. All performances are measured based on top-1 pose when taking 8 samples with the fine-tuned models. A. Median confidence of sampled points over at every iteration. Proportion of top-1 predictions below 2 over the course of the iterations for each cluster. C. Performance for APPPING D.
each cluster before the fine-tuning and after the K=60 steps of CONFIDENCE Aggregated performance along the fine-tuning for all the clusters weighted by their count with, as references, the performance some the baselines on the same set.
improvements either the score or confidence models will lead even further gains when finetuned with CONFIDENCE BOOTSTRAPPING.
CONCLUSION
Given the potential utility of high-accuracy blind docking in biology and drug discovery, it is impor- tant to track the progress of Ml--based methods generalize unseen pockets. To this end, we have proposed DOCKGEN, a new benchmark for blind docking generalization based on the classification of binding protein domains. Evaluating existing ML methods the DOCKGEN benchmark high- lights how overfitting training data prevents generalization unseen binding modes. By scaling the training data and model size well as integrating novel synthetic data generation technique, we were able significantly improve the generalization ability and developed and released DIFFDoCK- L, a new state-of-the-a docking method.
To improve generalization even further, proposed CONFIDENCE BOOTSTRAPPING, self- training method that only relies on the interaction between a diffusion and a confidence model and exploits the multi-resolution structure of the sampling process. This allows the direct fine-tuning of the docking model on classes of proteins where binding structural data not available. Empirically, the method shows significant improvements on the DOCK GEN benchmark, going from 10% to 24% success rate for efficient and fast models. Finally, we believe this opens up the possibility of training even larger-scale docking models that have so far been obstructed by the size and diversity of the available data, bringing step closer generalizable solution to the docking challenge.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
We thank Bowen Jing, Hannes Stark, Rachel Wu, Peter Holderrieth, Jody Mou, and Saro Passaro for their helpful discussions and feedback. We thank Jacob Silterra for his help in the code deployment.
This work was supported by the NSF Expeditions grant (award 1918839: Collaborative Research: Understanding the World Through Code), the Machine Learning for Pharmaceutical Discovery and Synthesis (MLPDS) consortium, the Abdul Latif Jameel Clinic for Machine Learning in Health, the DTRA Discovery of Medical Countermeasures Against New and Emerging (DOMANE) Threats program, the DARPA Accelerated Molecular Discovery program. the NSF AI Institute CCF- 2112665,, the NSF Award 2134795, and the GIST-MIT Research Collaboration grant. N.P. ac- knowledges support from NIH grant R00GM135519 and from the Innovation Research Fund of the Dana-Farber Cancer Institute.
REFERENCES
Tunde Aderinwale, Charles Christoffer, and Daisuke Kihara. Rl-mizerd: Multimeric protein docking using reinforcement learning. Frontiers in Molecular Biosciences, 9, 2022.
Amr Alhossary, Stephanus Daniel Handoko, Yuguang Mu, and Chee-Keong Kwoh. Fast, accurate, and reliable molecular docking with QuickVina 2. Bioinformatics, 31, 2015.
Mikhail Belkin, Daniel Hsu, Siyuan Ma, and Soumik Mandal. Reconciling modern machine- learning practice and the classical bari-ver trade-off. Proceedings of the National Academy of Sciences, 116(32): 0015518495555 2019.
H. Berman, K. Henrick, and H. Nakamura. Announcing the worldwide Protein Data Bank. Nat Struct Biol, 10, 2003.
Martin Buttenschoen, Garrett M Morris, and Charlotte M Deane. Posebusters: Ai-based docking methods fail to generate physically valid poses or generalise novel sequences. Chemical Sci- ence, 2024.
Hua Cheng, R Dustin Schaeffer, Yuxing Liao, Lisa N Kinch, Jimin Pei, Shuoyong Shi, Bong- -yun Kim, and Nick V Grishin. Ecod: an evolutionary classification of protein domains. PLoS compu- tational biology, 2014.
UniProt Consortium. Uniprot: a hub for protein information. Nucleic acids research, 2015.
Gabriele Corso. Modeling molecular structures with intrinsic diffusion models. arXiv preprint 1rXi::23022.22555. 2023.
Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, and Tommi Jaakkola. Diffdock: Dif- fusion steps, twists, and turns for molecular docking. arXiv preprint arXiv:2210.01776, 2022.
Justas Dauparas, Ivan Anishchenko, Nathaniel Bennett, Hua Bai, Robert J Ragotte, Lukas F Milles, Basile IM Wicky, Alexis Courbet, Rob J de Haas, Neville Bethel, et al. Robust deep learning- based protein sequence design using proteinmpnn. Science, 2022.
Valentin De Bortoli, Emile Mathieu, Michael Hutchinson, James Thornton, Yee Whye Teh, and Arnaud Doucet. Riemannian score-based generative modeling. urXiv preprint arXiv:2202.007663 2022.
Stefano Forli, Ruth Huey, Michael E Pique, Michel Sanner, David Goodsell, and Arthur Olson. Computational protein- ligand docking and virtual drug screening with the autodock suite. Nature protocols, 11(5):905 919, 2016.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative ad adversarial nets. Advances in neural information processing systems, 27, 2014.
Christoph Gorgulla, Andras Boeszoermenyi, Zi-Fu Wang, Patrick Fischer, Paul W Coote, Kr- ishna M Padmanabha Das, Yehor S Malets, Dmytro S Radchenko, Yurii Moroz, David A Scott, et al. An open-source drug discovery platform enables ultra-large virtual screens. Nature, 2020.
10

---

Published as a conference paper at ICLR 2024
Thomas A Halgren, Robert B Murphy, Richard A Friesner, Hege S Beard, Leah L Frye, W Thomas Pollard, and Jay L Banks. Glide: a new approach for rapid, accurate docking and scoring. 2. enrichment factors in database screening. Journal of medicinal chemistry, 2004.
Nafisa M. Hassan, Amr A. Alhossary, Yuguang Mu, and Chee-Keong Kwoh. Protein-ligand blind docking using quickvina-w with inter-process spatio-temporal integration. Scientific Reports, 7, 2017.
Liegi Hu, Mark L. Benson, Richard D Smith, Michael G Lerner, and Heather A Carlson. Binding moad (mother of all databases). Proteins: Structure, Function, and Bioinformatics, 2005.
Ajay Jain. Surflex: fully automatic flexible molecular docking using molecular similarity-basee -based search engine. Journal of medicinal chemistry, 46, 2003.
Wengong Jin, Jeremy Wohlwend, Regina Barzilay, and Tommi Jaakkola. Iterative refinement graph neural network for antibody sequence-structure co-design. arXiv preprint arXiv:21 1rXiv:2110.06624 2021.
Bowen Jing, Gabriele Corso, Jeffrey Chang, Regina Barzilay, and Tommi Jaakkola. Torsional diffu- sion for molecular conformer generation. arXiv preprini arXiv:2206.0172 2022.
Gareth Jones, Peter Willett, Robert C Glen, Andrew R Leach, and Robin Taylor. Development and validation of a genetic algorithm for flexible docking. Journal of molecular biology, 267(3):
727-748, 1997.
John Jumper, Richard Evans, Alexander Pritzel, Tim Green, Michael Figurnov, Olaf Ronneberger, Kathryn Tunyasuvunakool, Russ Bates, Augustin Židek, Anna Potapenko, al. Highly accurate protein structure prediction with alphafold. Nature, 596(7873):583-589 2021.
David Ryan Koes, Matthew P Baumgartner, and Carlos Camacho. Lessons learned in empirical scoring with smina from the csar 2011 benchmarking exercise. Journal of chemical information and modeling, 53, 2013.
Rohith Krishna, Jue Wang, Woody Ahern, Pascal Sturmfels, Preetham Venkatesh, Indrek Kalvet, Gyu Rie Lee, Felix Morcy-Burrows, Ivan Anishchenko, Ian R Humphreys, et al. Generalized biomolecular modeling and design with rosettafold all-atom. bioRxiv, pp. 2023-10, 2023.
Radoslav Krivák and David Hoksza. P2rank: machine learning based tool for rapid and accurate prediction of ligand binding sites from protein structure. Journal of cheminformatics, 10(1):1-12, 2018.
Zeming Lin, Halil Akin, Roshan Rao, Brian Hie, Zhongkai Zhu, Wenting Lu, Allan dos San- tos Costa, Maryam Fazel-Zarandi, Tom Sercu, Sal Candido, and Alexander Rives. Language models of protein sequences at the scale of evolution enable accurate structure prediction. arXiv,, 2022.
Tiqing Liu, Yuhmei Lin, Xin Wen, Robert N Jorissen, and Michael Gilson. Bindingdb: a web- accessible database of experimentally determined research, 2007.
binding affinities. Nucleic acids
Zhihai Liu, Minyi Su, Li Han, Jie Liu, Qifan Yang. Yan Li, and Renxiao Wang. Forging the basis for developing pigdein-ligand interaction scoring functions. Accounts of Chemical Research, 50, 2017.
Wei Lu, Qifeng Wu, Jixian Zhang, Jiahua Rao, Chengtao Li, and Shuangjia Zheng. Tankbind: Trigonometry-wwar neural networks for drug-protein binding structure prediction. Advances in neural information processing systems, 2022.
Heng Luo, Achille Makoue-Nkoutehe Nalini Singh, Lun Yang, Jianying Hu, and Ping Zhang. Molecular docking for prediction and interpretation of adverse drug reactions. Combinatorial Chemistry & High Throughput Screening, 2018
Matthew R Masters, Amr H Mahmoud, and Markus A Lill. Fasiondock: Physics-informed diffusion model for molecular docking. 2023.
11

---

Published as a conference paper at ICLR 2024
David McClosky, Eugene Charniak, and Mark Johnson. Effective self- self-training for parsing. In Proceedings of the Human Language Technology Conference of the NAACL, Main Conference, 2006.
Andrew T McNutt, Paul Francoeur, Rishal Aggarwal, Tomohide Masuda, Rocco Meli, Matthew Ragoza, Jocelyn Sunseri, and David Ryan Koes. Gnina 1.0: molecular docking with deep learn- ing. Journal of cheminformaties, 13, 2021.
Rocco Meli and Philip C. Biggin. spyrmsd: symmetry-correcte rmsd calculations in python. Jour- nal of Cheminformatics, 12, 2020.
Oscar Méndez-Lucio, Mazen Ahmad, Ehecatl Antonio del Rio-Chanona, and Jörg Kurt Wegner. A geometric deep learning approach predict binding conformations of bioactive molecules. Nature Machine Intelligence, 3(12):1033-1033 2021.
Mayukh Mukhopadhyay, A brief survey on bio inspired optimization algorithms for molecular docking. International Journal of Advances in Engineering & Technology, 2014.
Nicholas F Polizzi and William F DeGrado. A defined structural unit enables de novo design of small-moleeule-bininng proteins. Science, 369, 2020.
Gustavo Schottlender, Juan Manuel Prieto, Miranda Clara Palumbo, Florencia A Castello, Federico Serral, Ezequiel J Sosa, Adrián G Turjanski, Marcelo A Marti, and Darío Fernández Do Porto. From drugs to targets: Reverse engineering the virtual screening process on proteomic scale. Frontiers Drug Discovery, 2022.
David Silver, Aja Huang, Chris Maddison, Arthur Guez, Laurent Sifre, George Van Den Driessche, Julian Schrittiieser, loannis Antonoglou, Veda Panneershelvam, Marc Lanctot, et al. Mastering the game of go with deep neural networks and tree search. Nature, 529, 2016.
Yang Song and Stefano Ermon. Generative modeling by estimating gradients of the data distribution. Advances in Neural Information Processing Systems, 32, 2019.
Yang Song, Jascha Sohl-Dickstein, Diederik Kingma, Abhishek Kumar. Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. In Interna- tional Conference on Learning Representations, 2021.
Hannes Stärk, Octavian Ganea, Lagnajit Pattanaik, Regina Barzilay, and Tommi Jaakkola. Equibind: Geometrie deep learning for drug binding structure prediction. In International Conference on Muchine Learning. PMLR, 2022.
René Thomsen. Protein-ligant docking with evolutionary algorithms. Computational Intelligence in Bioinformatics, 2007.
René Thomsen and Mikael H Christensen. Moldock: new technique for high-accuracy molecular docking. Journal of medicinal chemistry, 49, 2006.
Chenran Wang, Yang Chen, Yuan Zhang, Keqiao Li, Menghan Lin, Feng Pan, Wei Wu, and Jinfeng Zhang. A reinforcement learning approach for bioinformatics, 23, 2022.
binding pose prediction. BMC
Joseph L Watson, David Juergens, Nathaniel R Bennett, Brian L Trippe, Jason Yim, Helen E Eise- nach, Woody Ahern, Andrew Borst, Robert J Ragotte, Lukas F Milles, et al. De novo design of protein structure and function with rfdiffusion. Nature, 2023.
Qizhe Xie, Minh- Thang Luong, Eduard Hovy, and Quoe V Le. Self-training with noisy student improves imagenet classification. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, 2020.
Xin Yi, Yanbo Zhang, Peidan Wang, Jiangwei Qi, Meiying Hu, and Guohua Zhong. Ligands binding and molecular simulation: the potential investigation of biosensor based insect odorant binding protein. International journal of biological sciences, 2015.
Gengmo Zhou, Zhifeng Gao, Qiankun Ding, Hang Zheng. Hongteng Xu, Zhewei Wei, Linfeng Zhang, and Guolin Ke. Uni-mol: a universal 3d molecular representation learning framework. 2023.
12

---

Published as a conference paper at ICLR 2024
A DOCKGEN BENCHMARK DETAILS
A. DATASET CREATION
In this section, we specify the details of how the Binding MOAD dataset was parsed and filtered to obtain the DOCKGEN benchmark validation and test sets. The benchmark was created following these steps:
1. We perform the ECOD-based clustering. Each ligand in protein is classified by the ECOD domain (1 group) of the chain that is making the most contacts with the ligand. The t-group of the ligand is assigned using the consensus of per-residue t-group assignments of each contacting amino acid (4.8A heavy atom distance) in the dominant chain. We use this approach to classify both the complexes in PDBBind and MOAD. There are 179 clusters in MOAD that do not appear in PDBBind. These 179 are randomly and equally divided between validation and test datasets.
2. Within these clusters, since most docking tools do not support the simultancous docking of multiple ligands, we discard ligands that are within 4.8A from the heavy atoms of another ligand.
3. We discard the ligands labeled by MOAD as "part of proteins" as these are constituted mostly by metals and covalent binders.
4. We discard very large ligands formed by more than 60 heavy atoms.
5. Ligands with equal chemical composition that are bound to the same protein are also clus- tered together forming single ligand with multiple correct poses.
6. To ensure diversity in the benchmark, if the same ligand appears bound to different biolog- ical assemblies of the same PDB entry we only select one these at random.
7. To avoid overrepresentation of certain ligand classes, we limit the maximum number of ligands to 5 of the same molecule separately in the validation and test sets. This avoids, for example, the presence of a large number of NAD ligands when considering the "NAD- binding domain" cluster.
8. To follow the convention that has been used to train previous ML-based blind docking algorithms, select only the protein chains that have C-alpha atom within 10Å of ligand's ' heavy atom. In the cases of multiple equal ligands bound to the same complex, one was selected as reference for the filtering.
After these steps we are left with validation set composed by 141 unique complexes from 70 different clusters and a test set formed by 189 complexes from 63 clusters.
A.2 DATASET ANALYSIS
In this section, we compare the test set of the newly generated DOCKGEN benchmark and the one of PDBBind.
Receptor and ligand dimension In Figure 5 we plot the distribution of the sizes of the receptors and ligands in the two sets. These are not too different, but on average DOCK GEN seems to present slightly smaller ligands but larger protein receptors. The latter might be one of the explanations for why even traditional search-base docking methods that are often considered generalize relatively well (even though they also present number of parameters fitted to experimental data) do worse in the DOCKGEN test set.
Binding site similarity analysis To evaluate binding site similarity between datasets, we extract residues in contact with the ligand (defined as protein's 's heavy atoms within 5 of ligand heavy atoms) and merge them into single chain to handle binding sites at protein-protein interfaces. We then TMalign every extracted binding site the training set to every extracted binding site in the test set for both the PDBbind and DOCKGEN datasets. We parse the T'allin output for any aligned residues and record their amino acid identities for both the target and query binding sites. We then compute the fraction of the target and query matched by the alignment and take the harmonic mean of those fractions a first metric for binding site similarity. We then use the BI BLOUMM62 amino acid
13

---

Published as a conference paper at ICLR 2024
Comparison in size of the ligands
Comparison in size of the receptor
0.08
POBBind test set
PDBBind test set
DockGen test set
DockGen test set
0.0030
0.07
0.06
0. 0.0025
0.05
0020
Normalzzed
0.04
0. 0.0015
0.03
0.00110
0.02
0.0005
0.01
0.00
0.0000
30
40
50
250
500
750
1000
1250
Number of Heavy Atoms in the Ligand
Number of Residues in the Receptor
Figure 5: Comparison of the sizes of the ligand and the receptor between the test sets of PDBBind and DOCKGEN.
substitution matrix to quantify similarity in the case that two amino acids are structurally aligned but not exact sequence matches. Amino acids that are more similar have larger values in this matrix. We normalize the BLOSUM matrix values between 0 and I and then take the average of these normalized values across all structurally matched residues as a second similarity metric.
We compare these metrics across the PDB and DOCKGEN test sets when compared to PDBBind training set in Figure and show that binding sites in PDBBind tend be more similar to those in the test set compared to the same comparison for the DOCKGEN dataset: the density in the top right corner indicates significant binding site overlap in PDBBind and little in the case of DOCKGEN.
B DATA AUGMENTATION STRATEGIES DETAILS
B. INCREASING THE TRAINING DATA
To increase the training data using the clusters of Binding MOAD that did not go into the validation or test set, we first take the clusters that remained after applying step I above. We also remove all complexes resolved in 2019 or later to maintain the validity of the PDBBind test set. Then, we again discard all ligands that are close to other ligands and those with single heavy atom.
We take the remaining complexes. forming a dataset with 20,012 ligand poses bound to 14,214 different biological units, and use them for training. For ligands that are bound to the same biological unit, at every epoch we select a single ligand at random for training. At training time, we simply concatenate this dataset PDBBind's 's training set. One should note that the datasets disjoint and many complexes will appear in both. We avoid removing the redundancy to give more weight to complexes that passed both filtering processes of MOAD and PDBBind and are therefore more likely to be of higher quality.
not
B.2 VAN DER MER- MER-INSPIRED TRAINING
To extract van der Mer-like amino-acid ligands, we start from large collection of protein structures that comprise the ProteinMPNN [Dauparas et al., 2022] training set. At preprocessing time, for every sidechain in the protein, we compute n, the number of other amino acids that have heavy atoms within 5Å and are not local in primary sequence (within 10 residues). This determines the extent to which the remaining part the protein forms a pocket around the selected sidechain, once the adjacent sequence on each side is removed.
At training time, we iterate over the clusters of proteins in the dataset. We use the same clusters adopted by ProteinMPNN that were generated with a sequence similarity cutoff of 30%. For each cluster, we take one protein at random, and within this, we select sidechain random with
14

---

Published as a conference paper at ICLR 2024
Visualization of the van der Mer-inspired synthetically generated docked poses. In this case, tyrosine (in red) taken be the ligand, and the amino acids that are nearby the primary sequence are removed from the protein structure of IQXZ (the created chain breaks are highlighted in orange).
Figure 6:
probability proportional to max(0, n. 5) sampling buried residues with a higher likelihood. If no sidechain more than 5 contacts then the protein is skipped.
Once a sidechain is selected, the sequence-local segment of the protein chain, consisting of 21 residues centered the selected sidechain (i.e. 10 residues either side), is removed from the protein structure. The sidechain and its backbone atoms are then used as the ligand that the model trained to dock. An example of such resulting complex is represented in Figure 6.
Note that for computational efficiency, instead of re-computing the embedding each time we truncate a protein structure, we compute the ESM embeddings [Lin et al., 2022] used by DiffDock only once with the full sequence and then simply crop the removed sequence out (even though the embedding of the remaining residues might be affected by the residues that were removed).
C CONFIDENCE BOOTSTRAPPING DETAILS
In this section, discuss the CONFIDENCE BOOTSTRAPPING procedure and experimental setup. We demonstrate that CONFIDENCE BOOTSTRAPPINS improves blind docking performance of the pretrained score model on a previously unseen cluster without knowing the ground-truth poses of that cluster. This is achieved by iteratively sampling from the score model (the rollout process), evaluating these samples with the confidence model, and updating the score model treating high- quality samples as ground-truth poses. Our procedure assumes a DIrEDOCK score model and confidence model both pretrained. We use the procedure outlined in Corso et al. [2022] to train and sample from the score model.
C.I TRAINING PROCEDURE
The training procedure takes as input ligand/receptor pairs D = { d2....) from cluster of in- terest, pretrained diffusion model parametrized by 8 with learned distribution over poses po(x; d) and score sy (x((")) t; d), and a pretrained confidence model (x, d). At each step. we sample com- plexes from the score model through the reverse diffusion process, score these complexes with the model. More specifically, we start with the buffer B that is initially empty: at step i, we sample g candidate poses (x,. d.) for j E {1.-.... g) for each d, D from Poi (x; d) and evalute these candidate poses with cz. We use the exponential moving averaged version of model weights 0i oime as in Corso al. [2022] to stabi- lize the inference over different training epochs. We add the sampled poses with a confidence score ca(x > ki, where l is the confidence cutoff, to the buffer: B = B U {(x,, de) (x(x,, d,) > k).
confidence model, and use high-confidence samples update the
15

---

Published as a conference paper at ICLR 2024
Then, update the score model # by treating complexes in B ground truth poses and performing a fixed number of SGD update steps on the diffusion objective to obtain 0i+1.
To train the original model we use the denoising score matching loss presented in Section 4.1 with a weighting A(t) = t/p t/2 where Me the expected score magnitude time t. During fine-tuning we use (1) 1/2 and (1) = L/14, to give more emphasis to the bootstrapping examples at high also be seen sampling & from different beta distributions while always normalizing by the expected score.
noise levels. Note that these different weightings
There are a few nuances to building and using the buffer B. First, we resample constant m. complexes from B to train the score model at every step. Motivated by the intuition that higher-score poses should be sampled more frequently, the poses are sampled from the distribution p(x, d) ~ exp Ga [x, d)] where Z is a normalization constant. Additionally, we enforce an upperbound on the number of samples per complex to encourage diversity across complexes in B by only keeping the n highest scoring poses per protein/ligand pair. Moreover, as alluded to in 4.3, updating the score model with samples from the buffer may cause it to lose information about how to perform steps with / < min and/or learn to sample from pathological spaces of the confidence model, where the confidence model assigns high scores to poses distant from the ground truth. Therefore, at each training step, we also include p randomly sampled complexes from MOAD training set in addition to complexes sampled from B. In our scheme, k; P. m, and n are hyperparameters selected through testing the method the validation dataset.
C.2 MODEL ARCHITECTURE
To enable the efficient execution of the CONFIDENCE BOOTSTRAPPING iterative training routine, make number of changes the architecture of DIFFDOCK's 's score (obtaining what we refer to as DIFFDOCK-S) and confidence models.
Score model We speed up the architecture and execution of DIFF score model in a number of ways. First, add a number of embedding message-passing layers which, unlike the cross- attentional interaction layers of DIFEDOCK, independently process the protein and ligand structures. This allows to increase the depth of the architecture with very little added runtime. In fact, due to the significantly higher number of nodes, the main complexity of the embedding layer lies in the protein component. However, under the rigid protein assumption, the structure is the same across all the different samples and diffusion steps. Therefore, the protein embedding can be computed only once resulting in minor computational overhead. Further, we limit the order of the spherical harmonics used represent the edges one (with minor loss in accuracy), and we batch and move the whole reverse diffusion process to GPU to improve parallelization and memory transfers.
Confidence model We also change number of design choices in the confidence model to make it more efficient and better suited for CONFIDENCE BOOTSTRAPPING. The first choice is force the model to reason about local interactions, as the affinity of a ligand to a particular pose is largely a local property. We achieve this by simply feeding to the model the receptor structures of only the amino acids whose C-alpha is within 20A of any of the predicted ligand atoms" positions.
Further, during the binary classification training of the confidence model, we try to remove the bias of the model against hard targets by balancing the proportion of positive and negative examples the model is trained over. When not possible with sampled poses (because the model only samples negatives), we use the so-called conformer-matede
ligand poses [Jing et al., 2022] as positive ex- amples. Moreover, to make the transition smoother and reduce the perceived variance in the labels, we separate the positive (poses with RMSD(22A and negative classes (>4A)', Finally, we super- vise the model not only to predict the label of the whole ligand but also that of each individual atom (supervised on their individual distance to the ground truth pose).
C.3 EXPERIMENTAL DETAILS
Setup. We tested CONFIDENCE BOOTSTRAPPING on DOCK Dlcclusssssssers, subset of DOCKGEN containing 8 clusters with at least 6 complexes each. For every cluster in DoCkGEN-clusters, we
WWe note that concurrent work Masters et al. [2023] also applies this strategy.
16

---

Published as a conference paper at ICLR 2024
started with the same pretrained diffusion and confidence models, and ran 60 iterations of CONFI- DENCE BOOTSTRAPPING, where each iteration contained a rollout step and an update step with 200 SGD updates. We did two evaluation runs with the same hyperparameters and reported the aver- aged results. To evaluate the generated complexes, we computed the symmetric-correcte RMSD of sPyRMSD [Meli & Biggin, 2020] between the predicted and the crystal ligand atoms, We reported the percentage top-ranked predictions that have an RMSD less than 2A, a metric generally con- sidered to be representative of getting the correct pose [Alhossary et al., 2015; Hassan et al., 2017; McNutt al., 2021; Corso et al., 2022].
Hyperparamettrs. In our experiments, we chose confidence cutoff A = -4, number of complexes sampled from PDBBind p- 100, number of complexes sampled from the buffer of inference samples = 32, and maximum samples per protein/ligand pair n 20. At the rollout step, we ran inference steps each with samples and compute the RMSD less than 2 metric with the top-ranked pose from each inference step reduce variance in the reported metric. Additionally, we set number of inference samples 80 for the first bootstrapping inference step fill the initially empty buffer. These parameters were selected by testing the method on the DOCKGEN validation clusters.
100, number
C.4 COMPUTATIONAL COST
The runtime cost of our fine-tuning approach depends on the number of complexes sampled and the number of gradient update steps. In our experiment, on average sample 320 complexes in the cluster for every 200 gradient update steps (with batch size 5). We interleave these two operations 60 times. With these parameters and training on one NVIDIA A6000 GPU, the average run time is hours.
One can try compare this cost with for example running GNINA (here we assume the default version without extra search exhaustiveness), which generalizes better without retraining but it is consider- ably slower and cannot be finetuned on specific domains. Assuming the methods (GNINA and DiftDock w/ confidence bootstrapping) work similarly, then one ask in which setting would it be faster to run either one them. It takes 232 inference complexes for DiftDock to ammortize the cost of the finetuning and from then onwards it can provide significant runtime improvements. As many screening campaigns require significantly more than 232 complexes, we believe that the idea of finetuning a docking model to the specific target class of interest may prove a useful feature in the future.
D CONFIDENCE BOOTSTRAPPING AND REINFORCEMENT LEARNING
In the section we discuss the connection of CONFIDENCE BOOTSTRAPPING to Reinforcement Learning (RL). Our objective can be approximately formulated finding some score model pa- rameters 8 to maximize the confidence conditioned on a cluster, and can be written as
8" argmax Jo
8
where
(I) This objective can be directly formulated as the RI. objective, where the reward function is the confidence model cs and the policy the diffusion model (x( d). Then, our method can be seen as loosely related to the policy gradient method, where the gradient of Equation can be written as
Jg = (xp( (x; d)].
V Jg V Exp (x:a) (cp(x;
(xu() V log pa (x; d)ca d)]
Vo log po xx; (x::
n2
In the policy gradient method, samples are collected to approximate the expected value, which corresponds to the number of poses we sample per complex during training. However, computing V log pe (x;; d) is intractable for diffusion models. Instead, we modify the reward to exp(cs (x;(cs(x()) and the objective to:
17

---

Published as a conference paper at ICLR 2024
0x = argmax (ppxxx))eepp(caa d)) log Po (x)]
0
argmin (xxd) exp(cx(Xxd)) (xx))))) log po (x)
argmin
(o))
Again this cannot be estimated directly but can the denoising score matching loss with Per o(x, d) as our data distribution to derive an upper bound that can be optimized [Song & Ermon, 2019]:
Epp o (x.d) 1- log lo (x)) - JDSM (0) +C
where C' is a constant that does not depend on 0 and InSM (0) is the denoising score matching loss.
Avoiding overoptimization Note that unlike traditional RL problems we not only have the (de- noising score matching) loss based on the reward from our confidence model, but we also keep the loss component that came from our training set optimization. While the RL objective is directly maximize reward/confidence, we also want to prevent the model from overfitting into pathological spaces of the confidence model (where the confidence model gives high confidence for bad poses), aka "overoptimization" which is why we included the real samples from training set.
RL in docking Reinforcement learning has been applied in the past for the docking task, for example, Wang et al. [2022] uses it for protein-ligand docking while Aderinwale et al. [2022] applies it to multimeric protein docking. These works have two strong differences with approach: (1) RL used to optimize the pose at inference time for specific complex not to train/fine-tunc model (2) the specific methodologies used are very different from the idea of CONFIDENCE BOOTSTRAPPING.
E ADDITIONAL. EXPERIMENTS
E.1 POSEBUSTERS BENCHMARK
We report in Table the results of DIFFDOCK-L on the PoseBusters benchmark test set compared to the recent SOSETTAFOLD-ALLATEE
[Krishna et al., 2023] and all the methods reported by But- [2024]] GOLD [Jones al., 1997], VINA [Forli al., 2016], DEEPDOCK [Méndez- Lucio et al., 2021], UNI-MOL [Zhou et al., 2023], EQUIBIND [Stärk et al., 2022], TANKBIND [Lu al., 2022] and DIFFDOCK [Corso et al., 2022]. Even this benchmark DIFFDOCK-L, performs best among blind docking methods.
tenschoen
E.2 CONFIDENCE BOOTSTRAPPING ORACLE EXPERIMENTS
In separate experiment, we tested CONFIDENCE BOoTSTRAPPING with "oracle" confidence pre- dictions. The oracle confidence predictor is a monotonic transformation of the true RMSD between the ground truth pose and the predicted pose: ca(x, d) = -4 tanh SRRSSD(xxx - -2). Note that this is not a practical setting as we would not have access to the ground truth poses x" in real applications. However, this is illustrative experiment establishing upperbound on the perfor- mance gains achievable through CONFIDENCE BOOTSTRAPPING with a "perfect" confidence model (Figure 7).
E.3 CONFIDENCE BOOTSTRAPPING ABLATION STUDIES
To better understand the benefits of different components of CONFIDENCE BOOTSTRAPPING, we conducted ablation experiments and report the ablation results (Figure 8) on the DOCKGEN vali- dation dataset. This validation dataset is selected by choosing all clusters with four or more com- plexes in DOCKGEN, excluding those in DOCK DoCKGEN-Clin GEN-clustess. We further exclude one cluster where DIFFDoCK-L -L already achieves higher than 80% RMSD less than 2, and another where an insuffi- cient number of samples pass the confidence threshold, resulting in 7 clusters. Here, we provide a description of each ablation experiment.
18

---

Published as a conference paper at ICLR 2024
Table 2: Comparison of the performance the PoseBusters dataset. Pocket-hased docking methods receive further information about the location and shape of the binding pose. unlike the other blind docking methods reported ROSETTAFOLD- FOLLTTFOLL-LLOTOO ALLATOM (1) does not take imput the holo-structure of the protein, (2) does take as imput the information about the cofactors involved in the binding, and (3) uses training cutoff date April 2020 instead of December 2018.
Method
RMSD 2À
Pocket-based docking
GOLD
58%
VINA
60%
DEEPDOCK
20%
UNI-MOL
22%
Blind docking
EQUIBIND
2%
TANKBIND
16%
DIFFDOCK
38%
FOLLTTAAFLL--ALLATTOO
42%
DIFFDOCK-L
50%
Temperature 0 Instead of sampling from the buffer with p x exp[ca(x, d)), which can be inter- preted as x explkea(x, (x, d)] with temperature & = 1, we investigate the effects of drawing buffer samples with uniform probability 0).
1 sampling We change the distribution of / E [0, 1]. the amount of noise added to each sample during training time. Heuristically, would prefer to sample higher values of more often to finetune the earlier stages of the reverse diffusion process, guiding the model into finding the right pocket. is sampled from beta distribution Beta(o, B). In addition, samples from Prnin use different value of a and |3 from buffer samples. CONFIDENCE BOOTSTRAPPING, we have a = 2, 3 = 1 for buffer samples, and = 1. 3 for Puain samples, with tmin O. We experiment with different variations of these default parameters (Figure 8).
Removing samples from Ptrain In this experiment, we remove Puain complexes (real samples) from the update steps of the diffusion model, using only samples from the buffer. These complexes were initially added to avoid the overoptimization problem as discussed previously, and removing them, unsurprisingly, decreases the performance of our method (though our method still significantly outperforms the baseline DiffDock).
19

---

Published as a conference paper at ICLR 2024
100
10
Bacterial fluorinting enzyire...
Median Oracle Confidence Score
FAD binding domain
bon---ii hydrolase
Molybdenum cofactor besynthesis...
6
Nicleatide-Sincing
Hamo oligamenic ttvin-containing...
RMSD (%)
4
SGNH hydrolasz
Glutaminase/kspragggene
0
-2
20
-6
10
20
30
60
10
20
30
40
50
60
A.
Bootstrapping Iterations
8.
Bootstrapping Iterations
100
100
Boatstrapping
After Booosstrapingg
RMSD 2À (%)
RMSD 22 (%)
20
20
Oracle Fine-tuned Dif'Dock
Fine-tuned DitDDckk
0
2
10 20
30
40
50
60
Test Clusters
D.
Bootstrapping Iterations
Figure 7: Empirical performance of CONFIDENCE BOOTSTRAPPING with oracle confidence across the 8 protein domain clusters within DOCKGEN-cluster the bootstrapping method is run twice for every cluster and we show the average results of the two runs. All performances are measured based on the top-1 pose when taking 8 inference samples with the fine-tuned models. A. Median confi- dences of sampled points every iteration for each cluster. B. Proportion of top-1 predictions below 2Á over the course of the iterations for each cluster. C. Performance for each cluster before the fine- tuning and after the K=60 steps CONFIDENCE BOOTSTRAPPHES D. Aggregated performance all the clusters weighted by their number of complexes, showing results using the oracle confidence model and pretrained confidence model.
20

---

Published as a conference paper at ICLR 2024
Temperature
Buffer sempling alpha m
30
Confidence Bootstrapping
Training t sampling alpha -
Buffer sampling
minimum 0.3
25
No training samples
RMSD 2A (%)
20
15
10
20
30 Bootstrapping Iterations
50
60
A.
Before Boulstrapping
Temperature
Ruffer sampling alpha
Contidance Bootstrapping
Training : sampling alpha -
Buffer sampling alpha minimum
training samples
RMSD < 2Á (%)
20
10
All complexes
8-bladed
Acyi-CoA dehydrogenase C-terninal donain--nni iiliikkk
Aminomethyltraee feraee binddnng domain
Barrel domain heme- dependent catalases
L-sulfol actate dehydrogenase-llke
Phosphofructoin Sonpppofuccokiiaseeeee N terminal domain
Ribonucleas wwwuulelseeeeco Rh-llike
Figure 8: Ablation studies on CONFIDENCE BOOTSTRAPPING tested on the protein domain clus- ters within the validation set. All performances are measured based on the top-1 pose when taking 8 inference samples with the fine-tuned models. No Real Samples refer the algorithm using only buffer samples in the loss, and Alpha=3 refers to the altered distribution of noise schedule during training. A. Aggregated performance for all the clusters weighted by their number of complexes, showing results for the different ablation experiments. B. Performance for each cluster before the fine-tuning and after the K=60 steps of CONFIDENCE BOOTSTRAPPING.
21


Reviews:
[{"review_id": "ZIfi3usXuy", "reviewer": "Reviewer_ksBz", "summary": "A new docking benchmark called DockGen is introduced in this paper. Molecular docking methods based on machine learning do not perform well when confronted with unknown binding pockets, and this issue is not adequately addressed in previous benchmarks. In this benchmark, the validation and testing datasets are extracted from a different database, Binding MOAD, which contains new classes of binding pockets (based on clustering). The testing binding pockets are ensured to have a different shape than those in the training dataset. The ML-based docking methods fail in this benchmark, so a new training strategy is proposed to improve the results of the diffusion-based models. This strategy is called Confidence Bootstrapping and can be used to train (or fine-tune) DiffDock. The paper demonstrates that this methodology is effective in improving the results of DiffDock in the proposed benchmark, which focuses on model generalization.", "questions": "1. Why have you decided to call your method Confidence Bootstrapping? This setup does not remind me of the classical bootstrapping method. The sampling is done from the continuous distribution (not a discrete distribution with replacement), and each sample is processed independently (not resampled as a subset of the population sample). Why is this method not called, e.g., Confidence Sampling?\n2. What do you think about including conformation quality metrics in the benchmark, like those proposed in PoseBusters? Do you think that self-training using the confidence model can impact the pose quality in any significant way?\n3. Based on the description in the paper, I was not sure which dataset was used for fine-tuning. I checked in the code, and `DistillationDataset` uses the Binding MOAD data. Could you confirm that? (see also the last point in Weaknesses)", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The motivation of this paper is very clear, and new benchmarks for ML-based docking are needed.\n- The new benchmark is sourced from a different database than used in previous benchmarks, and the data is carefully curated.\n- Poor performance of the known ML-based docking methods is proven using the new benchmark.\n- A new method for training diffusion models is proposed to mitigate the problems with generalization.\n- The proposed Confidence Bootstrapping method achieves good results for the introduced benchmark.\n- Confidence Bootstrapping is implemented for DiffDock, where some architectural components (score and confidence models) were optimized.\n- Two ways of expanding the training datasets are proposed, and they lead to better testing results.\n- The benchmark and code were published along with this paper.\n\nOverall, the idea behind the paper is clear, and the propositions are original and significant for the field of study (improving ML-based molecular docking).", "weaknesses": "- The mathematical formulas in the paper should be corrected. Some parentheses are missing, and there are some symbols that were not introduced in the text, e.g. $p_{0t}$. I understand that the equation in Section 4.1 should correspond to the equation introduced in the cited publication (Song et al., 2021), but the new undefined symbols and missing parentheses make this equation hardly readable.\n- In Table 1, the results of DiffDock with modification explained in Section 4.4 and without dataset extension should also be presented.\n- In this benchmark, the only evaluation metric (besides computation time) is the percentage of the poses with RMSD below 2 or 5 A. Given the recent criticism of ML-based docking models, it would be advised to include conformation quality metrics like those proposed in PoseBusters [1].\n- It would be interesting to quantify the similarity between the binding pockets used in training, fine-tuning, and testing. I am curious if using complexes from the same dataset for fine-tuning and testing could create biases in the data due to the way the data is preprocessed and filtered. Furthermore, you should also try fine-tuning using only PDBBind structures, and see if the results for the DockGen benchmark improve in this setup.\n\n[1] Buttenschoen, M., Morris, G. M., & Deane, C. M. (2023). PoseBusters: AI-based docking methods fail to generate physically valid poses or generalise to novel sequences. arXiv preprint arXiv:2308.05777.\n\nBefore this paper can be published, it is essential to clarify the mathematical formulation of the method. Furthermore, additional results that disentangle the three new factors (architectural changes, dataset expansion, and confidence bootstrapping) would not only increase the credibility of the findings but also enhance the overall quality of the paper. I am willing to increase my score if my comments are properly addressed by the Authors.\n\n-----------\nEdit: I changed my score (5 -> 6) after reading the other reviews and Authors' responses. The paper was significantly improved. Most of my concerns were resolved, and it seems the remaining concerns can be resolved in the final version of the manuscript if the running experiments are finished.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BqgzDOniLw", "reviewer": "Reviewer_kTFp", "summary": "The authors address the general question of assessing and improving generalization of blind docking.\nTheir contribution are two-fold.\nFirst, they establish a new benchmark for blind docking, coined DockGen, aiming at assessing better the generalization capabilities of docking methods to unseen binding pockets.\nThis benchmark is mostly built upon building new validation and test sets, composed of complexes with binding pockets that are not present in the \"classic\" training set of PDBBind.\nUsing this new benchmark, they show that there exist an important generalization that existing methods, especially ML-based ones, fail at bridging (and which was largely invisible with the classic PDBBind benchmark).\nSecond, after showing that classic data augmentations strategies fail at bridging this gap, they suggest a new training paradigm, coined Confidence Bootstrapping, that aims at leveraging the feedback from a confidence model to guide the diffusion of a pre-existing (although modified for this purpose) method called DiffDock.\nAfter justifying the approach and formalizing it, the authors benchmark their bootstrapping method, showing great improvement over the already published DiffDock it is based on (and other methods).", "questions": "In addition to comments / answers to the points I raised in the weaknesses section, I have very few extra additional question.\nThe choice, during the update step, of using a ratio of half-half training samples coming from real, and from the buffer seems a bit arbitrary to me, especially combined with the choice of using only training samples for small t.\nWouldn't it make sense to use a ratio that varies more smoothly, i.e. an increasing proportion of buffer samples as t increases?\n\nAlso, the results in Table 1 look at the percentage of top-ranked predictions with RMSD smaller than 2 and 5A.\n2A certainly is a commonly used and satisfactory threshold.\nOn the other hand, 5 is admittedly a very high threshold (cf the remark at the end of Section 4 stating that a RMSD larger than 4A constitutes a negative example).\nDo you have any comment about this?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "The first contribution itself (DockGen) is already very valuable in itself as it sheds light on some of the shortcomings of current ML-based docking methods.\nWhile it is very common that widely used benchmarks in the biomedical field fail at capturing the real (and meaningful) difficulty of the task at hand, it is always a great contribution when this comes to light and when a novel benchmark is suggested to address it.\nThe use of the ECOD classification to cluster complexes in a more \"meaningful way\" than the more traditional chronological or sequence similarity -based approach is well motivated and using separate external complexes from MOAD allows to maintain the training set of PDBBind that everyone has been using in the field, making it easy for researchers of the field to use DockGen to take a deeper look on the generalization properties of their method.\nThe results coming from that benchmark give interesting insights on further limitations of current ML-based docking methods.\nDespite the very positive results that Confidence Bootstrapping obtains on that benchmark, it does not give (by any mean) the impression that it has has been established for the sole reason of making their new method \"look good\" (which can sometimes happen).\n\nThe second contribution is also well-motivated by well-known yet crucial observations on the nature of the docking task (i.e. it is hard to generate a good pose but \"easy\" to check if a pose is good).\nBefore jumping into its formalization, the authors do a very good job at explaining their approach on a higher level, risking some insightful analogies to seemingly loosely related approaches (e.g. AlphaGo), making the reading of the paper a very pleasant experience.\nDespite the importance of the contributions and the very limited length of the appendices, the authors also make a good job at giving enough details to extensively describe their approach and allow to reproduce the results.\n\nLast but not least, the results obtained by the new bootstrapping strategy seem to produce very encouraging results in terms of generalization to unseen clusters, although it is still completely failing at generalizing to some of the clusters.\nIn fact, the results seem so good that they may offer a proper, practical alternative to search-based approaches.", "weaknesses": "One main concern I have is related to the relationship of the current manuscript with the already published DiffDock.\nWhile the Confidence Bootstrapping is very specific to diffusion models and clearly builds on top of DiffDock, the authors have made a number of somewhat significant changes to it, even before testing the Confidence Bootstrapping (i.e. the results in Table 1).\nI think it makes things a bit confusing for readers that are already very familiar with the field and the method (which I was not).\nOn a side note, while I do not think that the anonymity of this work has been technically breached, I think that this lack of clear separation between (Corso et al. 22) and the current manuscript gives hints that they are likely to have been authored by the same people.\n\nAlthough the writing and presentation is overall very good, I find a few small parts to lack clarity.\nFor instance in Section 4.2 (paragraph 4), it is not really clear to me why the more local and easier task of testing the goodness of a pose can make it easier to generalize to unseen targets (although the experimental results sure seem to suggest so).\nThe relationship to the multi-resolution structure of diffusion models is also not very clear to me.\nI understand that the manuscript is already dense, but in Section 4.4 the explanation and justification on the choice to limit the order of the spherical harmonics is a bit cryptic, especially compared to the level of details of the rest.\n\nTo get a clearer overall picture of the performance of the proposed approach(es), I think it would make sense to add the Confidence Bootstrapping results in Table 1 (even if results would only be reported in the DockGen-clusters column).\nIn addition to be able to see more clearly the improvement in terms of performance, it would also be useful to see the runtime of this approach.\nI have no insight on how costly the confidence bootstrapping is, from a runtime perspective.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "XZQwqdnSeQ", "reviewer": "Reviewer_f6Xk", "summary": "This paper proposes a new benchmark and training method for evaluating and improving the generalization ability of machine learning models for molecular docking. The key points are:\n\n- Existing docking benchmarks are limited in diversity, focusing on common binding modes. A new benchmark DOCKGEN is proposed based on binding protein domains to better test generalization.\n- ML docking models show very weak generalization on DOCKGEN. Increasing training data and data augmentation do not help much.\nA new training approach \"Confidence Bootstrapping\" is proposed. It trains a diffusion docking model using high-confidence poses judged by a separate confidence model, without ground truth data.\n- Confidence Bootstrapping significantly improves model accuracy on DOCKGEN, doubling search-based methods. It is more sample-efficient than self-training.", "questions": "NA.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- DOCKGEN is a valuable new benchmark to rigorously test model generalization in docking. I really like this work about this kind of new evaluation of the current dataset problem and create a new split setting.\n- Thorough experiments demonstrate current ML methods generalize poorly, motivating new approaches.\n- Confidence Bootstrapping is innovative, exploiting diffusion model structure and confidence feedback.\n- Strong gains shown on DOCKGEN highlight its potential for advancing blind docking.", "weaknesses": "- Confidence boostrapping is much like a RL-based method, therefore the authors need to add more discussions about RL and its related works in docking and science. \n- There is one recent work that achieves strong performance on unseen targets (though split by UniProt ID), the authors are strongly encouraged to compared the performance on that model: FABind, https://arxiv.org/abs/2310.06763\n- More analysis needed on what protein features DOCKGEN probes that past benchmarks missed. \n- Ablation studies could better isolate benefits of different components of Confidence Bootstrapping. \n- Computational cost and sample efficiency could be compared to alternate training schemes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yEI9SYvuES", "reviewer": "Reviewer_Eb79", "summary": "The paper proposed a new benchmark in the field of docking. Under more strict data split, the authors claimed that currently developed deep learning methods failded to generalize to unseen proteins. A series of efforts were made to tackle such limitations, including architecture modifications, increasing the training data, Van der Mer-inspired docking and most important confidence bootstrapping. However, despite of a brilliant research idea and meaningful topic, the paper failed to justify their contributions with convincing and reproduciable experiments.", "questions": "- Why is ECOD used for data split? Best to my knowledge, SCOP(e) and CATH are more widely acknowledged.\n- 1QXZ and 5M4Q have a sequence similarity of 22% but not 16%. I blast it myself.\n- In Table 1, do you retrain GNINA or other machine learning methods under the DockGen split?\n- Why do traditional methods like SMINA not perform well on the DockGen benchmark?\n- Only 4 out of 8 clusters are proved to be improved by confidence bootstraping. Why are other clusters not improved?\n- In the paper Do Deep Learning Models Really Outperform Traditional Approaches in Molecular Docking? (https://arxiv.org/pdf/2302.07134.pdf), a new baseline of DIFFDOCK+Vina(UniDock) is proposed. It would be more approperate to be compared with than P2RANK+SMINA.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "- The DockGen Benchmark provides a new perspective to the evaluation of ML docking methods. Taking the similarity of binding pockets into account is more reasonable.\n- Confidence bootstrapping provides a new data-free methodology to improve docking models on unseen proteins.", "weaknesses": "- The paper mentioned a lot of works in terms of neural network architecture and data augmentation. However, I could not see their connections to the main topic of generalization. In Table 1, all those attempts failed to improve DIFFDOCK performance in the DockGen benchmark in terms of success rate(<2A). \n- Although a code repo is provided, I can not find any details about how baselines are benchmarked with the DockGen dataset. Specifically, when new data-split is provided, it is important to report whether baselines are re-trained or not. I am also confused by unclearified marks like EX. 64\n- The effect of confidence bootstraping is not approperately evaluated. Although the performance is demonstrated on 8 clusters, it is documented that there are 63 clusters in the test set. It is not convincing to claim the proposed method is effective before a more systematic and quantitative evaluation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "T3i2lZ1D4W", "reviewer": "Reviewer_Jhgj", "summary": "A new benchmark and a novel training paradigm have been developed to enhance generalization capabilities. The effectiveness of the model has been demonstrated through experiments.", "questions": "I don't have any further questions at the moment.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The experimental section is comprehensive and well-designed. The proposed model demonstrates good performance on specific tasks.", "weaknesses": "Due to my current level of expertise, I have not identified any shortcomings in this paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)