KinDEL: DNA-Encoded Library Dataset for Kinase Inhibitors

Authors: Benson Chen, Tomasz Danel, Patrick J. McEnaney, Nikhil Jain, Kirill Novikov, Spurti Umesh Akki, Joshua L. Turnbull, Virja Atul Pandya, Boris P. Belotserkovskii, Jared Bryce Weaver, Ankita Biswas, Dat Nguyen, Gabriel H. S. Dreiman, Mohammad Sultan, Nathaniel Stanley, Daniel M Whalen, Divya Kanichar, Christoph Klein, Emily Fox, R. Edward Watts
Primary area: datasets and benchmarks
Keywords: DEL, small molecule, benchmark, dataset

Abstract:
DNA-Encoded Libraries (DEL) are combinatorial small molecule libraries that offer an efficient way to characterize diverse chemical spaces. Selection experiments using DELs are pivotal to drug discovery efforts, enabling high-throughput hit finding screens. However, limited availability of public DEL datasets hinders the advancement of computational techniques designed to utilize such data. To bridge this gap, we present KinDEL, one of the first large, publicly available DEL datasets on two kinases: Mitogen-Activated Protein Kinase 14 (MAPK14) and Discoidin Domain Receptor Tyrosine Kinase 1 (DDR1). Interest in this data modality is growing due to its ability to generate extensive supervised chemical data that densely samples around select molecular structures. Demonstrating one such application of the data, we benchmark different machine learning techniques to develop predictive models for hit identification; in particular, we highlight recent structure-based probabilistic approaches. Finally, we provide biophysical assay data, both on- and off-DNA, to validate our models on a smaller subset of molecules. Data and code for our benchmarks can be found at: https://kin-del-2024.s3.us-west-2.amazonaws.com/kindel.zip

Decision: reject
Reviewer scores: 6, 5, 5, 6

Reviews and discussion:
<thread>
creation date:
1729232345127

thread_type:
official_review

<thread_object>
creator: reviewer_63rq

<content>
summary:
This paper propose a new open-source dataset as well as related benchmark for the DEL community. The main motivation behind this paper are: 

1. DEL community lacks a large, publicly available DEL dataset to benchmarking tasks. 
2. Current DEL dataset contains large bias and noise, and the existing methods cannot greatly address this issue. 

So, the authors propose an open-source dataset and a related enhancement approach to address these challenges. In details, these improvements and contributions are: 

1. KinDEL: a library of 81 million small molecules tested against two kinase targets, MAPK14 and DDR1, which is novel and with large amount. 
2. A comprehensive benchmark tested on current computational methods with on both on-DNA and off-DNA settings.

The proposed dataset and benchmark have great potentials to stimulate the development of the community.

strengths:
1. The dataset constuction process is reasonable, sound, and comprehensive. Also, the related process is explained clearly.
2. The corresponding evaluation to the proposed dataset is comprehensive, and the two types of splits "random" and "disynthon" enhance the perspective of the benchmark. 
3. The evaluation is clear and straightforward (Table1, Table2, and Figure6). 
4. The comparison to the current datasets is well-written and shows the valuable insights from authors and their advanced understanding to the DEL-related tasks.

weaknesses:
1. While there are very interesting performance comparison in the shown Table (Table1 and Table2), the explaination of the experiment results are expected, such as "why RF method performs the best on on-DNA set (Line 335)", "why there are different performance rankings on on-DNA and off-DNA settings?" and "why DEL-Compose(M) and DEL-Compose(S) performs differently on on-DNA and off-DNA settings?". I believe the insight provided by the authors would make the benchmark more solid and comprehensive. 
2. The proposed experiments including general performance (Table1, and Table2), and the visualization of experimental replicates are somehow not comprehensive enough. Serving as a benchmark for the DEL-community, more views and new settings are required, such as case study of top-ranking candidates (which can be potential candidates for real-world application), subset-Spearman coefficient (which is proposed in the paper "DEL-Dock: Molecular Docking-Enabled Modeling of DNA-Encoded Libraries" [1]), and potential effect of chemical properties and data source selections (building blocks) to the method performances on the KinDEL dataset. Then, the multi-view perspective of the benchmark could lead to larger contribution to the community. 

Reference:
1. Shmilovich K, Chen B, Karaletsos T, et al. DEL-Dock: Molecular Docking-Enabled Modeling of DNA-Encoded Libraries[J]. Journal of Chemical Information and Modeling, 2023, 63(9): 2719-2727.

questions:
1. According to Table1 and Table2, the SOTA performance w.r.t. Spearman coefficient can reach over 0.7, which is a very promising result,  but in paper "DEL-Dock: Molecular Docking-Enabled Modeling of DNA-Encoded Libraries" [1], there's an another proposed dataset containing Fingerprint and docking poses, where existing methods can only achieve relatively poor performance (around 0.30 w.r.t. negative Spearman coefficient) compared to KinDEL dataset. May the authors explain what is the inner differeneces between two datasets that leads to the obvious differences between two datasets? I also believe this comparison can make this work more solid as the contributing benchmark. 
2. Is it possible to provide an advanced version of KinDEL dataset with machine-aided of molecule docking poses. I understand it's very time-consuming and CPU-resource costly, but the DEL dataset with 1D and 3D modalities would lead to wider applications and evaluations to this community. Considering the potential cost, I believe it is also great to have dataset with only fingerprint (1D) information of the molecules. 

Reference:
1. Shmilovich K, Chen B, Karaletsos T, et al. DEL-Dock: Molecular Docking-Enabled Modeling of DNA-Encoded Libraries[J]. Journal of Chemical Information and Modeling, 2023, 63(9): 2719-2727.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730289511064

thread_type:
official_review

<thread_object>
creator: reviewer_6jas

<content>
summary:
The paper presents KinDEL, one of the first large publicly available DNA-Encoded Library (DEL) datasets focused on kinase inhibitors, specifically targeting MAPK14 and DDR1 kinases. The papers benchmarks different machine learning methods for binding prediction.

strengths:
1. the paper is clearly written and contains the experimental details in the appendix.

2. The dataset includes off-DNA data. The benchmark used different data split strategies.

weaknesses:
1. This paper is a valuable contribution to DEL-based drug discovery. It may serve as a good resource for computational drug discovery. But I am not sure about the importance of this paper for the ICLR community. 

2. The dataset has only two targets and both are kinase. The biophysical assay validation set is relatively small.
3. The authors did not mention the library size or sequence depth of the DEL dataset. Does it have an effect on the dataset?
4. The authors show that DEL-Compose performs better for off-DNA data. It would be helpful to discuss the potential biases due to the DNA barcode.
5. Why does the RF method become worse for the disynthon split?

questions:
See Weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730719105590

thread_type:
official_review

<thread_object>
creator: reviewer_bhqp

<content>
summary:
This study introduces a dataset of DNA-Encoded Libraries (DEL) focused on two specific kinases: Mitogen-Activated Protein Kinase 14 (MAPK14) and Discoidin Domain Receptor Tyrosine Kinase 1 (DDR1). Although the DEL datasets have proven valuable in drug discovery, they are relatively scarce for public use. The introduced dataset, named KinDEL (Kinase Inhibitor DNA-Encoded Library), comprises 81 million small molecules tested against MAPK14 and DDR1 kinases. An experimental evaluation is provided, comparing the performance of the proposed method in both on-DNA and off-DNA scenarios.

strengths:
+ The availability of a well-curated and publicly accessible dataset is a notable contribution on its own, making this work a valuable resource for the research community.

+ The authors have conducted a thorough review of the relevant literature, effectively establishing the originality and motivation of the study, and demonstrating a clear understanding of the current state of the field.

weaknesses:
- Given that the primary contribution of this work is a dataset, it would be beneficial to evaluate state-of-the-art (SOTA) methods on it, both to assess their performance in a new context and to demonstrate the dataset's comprehensiveness. The methods tested seem mostly old ones.

questions:
- Referring to Section 5.1 Current Datasets, there have been, especially recent, efforts of providing DEL datasets even though they don't exactly match the features offered by the present study. However, it might possible that these existing datasets could be adapted to resemble KinDEL.Could you elaborate on whether KinDEL is novel in that sense, i.e., for instance, enhancing the dataset from [Iqbal et al. (2024)] by incorporating on-DNA synthesis can be challenging?

Sumaiya Iqbal, Wei Jiang, Eric Hansen, Tonia Aristotelous, Shuang Liu, Andrew Reidenbach, Cerise Raffier, Alison Leed, Chengkuan Chen, Lawrence Chung, et al. DEL+ ML paradigm for actionable hit discovery–a cross DEL and cross ML model assessment. ChemRxiv
doi:10.26434/chemrxiv-2024-2xrx4, 2024.

- Furthermore, how does KinDEL compare to the existing DEL datasets in terms of diversity, and how well does it reflect the performance of existing methods in predicting Poisson enrichment? Does KinDEL offer a more comprehensive or representative testbed for evaluating these methods?"
</content>
</thread_object>

</thread>
<thread>
creation date:
1730721212754

thread_type:
official_review

<thread_object>
creator: reviewer_z3ms

<content>
summary:
The authors have released a new dataset, KinDEL, based on DNA-encoded library (DEL) testing, specifically targeting two kinases, MAPK14 and DDR1. They conducted experiments on this dataset to test model performance.

strengths:
The authors provide a substantial amount of new data.
The structure of the article is clear and easy to follow.

weaknesses:
1. The evaluation is limited to only two kinase targets, MAPK14 and DDR1. Given that both targets are kinases, this dataset may have limited generalizability as a benchmark for models applied to broader, non-kinase targets.
2. The data-splitting method may ensure that compounds with the same disynthon do not end up in the same split, but it doesn’t fully prevent similar compounds from being grouped together, as disynthons do not necessarily represent the core structure of small molecules. Other approaches, such as scaffold-based or overall molecular similarity-based splits, may yield a more robust assessment.
3. Presentation Improvements: The table headers are somewhat confusing, making it unclear what the numbers in the table represent without reading the text. In Figure 3, "SP³" should be corrected to "sp³" for accuracy.

questions:
1. Why use AI to reduce data noise? If DEL diverges significantly from reality, it may indicate instability or unsuitability for the current task. AI-based discriminative models are inherently inaccurate to some extent, so how effective is it to use one inaccurate method to adjust for another?
2. More explanation is needed for why certain chemical properties in Figure 3 are considered "drug-like". For instance, the molecular weight peak exceeds the traditional threshold of 500, and the QED values are relatively low.
3. Why is on-DNA data significant here, when off-DNA structures are more relevant for practical applications like drug development? On-DNA structures are unlikely to be developed as drugs.
4. Why didn’t the authors test more end-to-end models? Also, why did they use Morgan fingerprints as input instead of molecular representations like SMILES strings (1D), molecular graphs (2D), or atomic coordinates (3D)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732148203723

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
(General Response) Thank you for your feedback!

comment:
We would like to extend our gratitude for your invaluable feedback. Below, we have summarized the main concerns shared by the reviewers and how we want to address them:

**1. Only two kinase targets are included.**

We recognize the importance of increasing target diversity to make our benchmark more widely applicable, especially for methods tailored to different biological targets. Nonetheless, we firmly believe that our KinDEL dataset already serves as a significant resource for the development of new machine learning techniques focused on analyzing large combinatorial libraries. We have chosen two biological targets that not only complement existing public DEL datasets but are also well-studied and established within the scientific community (see for example the studies on the identification of MAP Kinase inhibitors confirmed with crystal structures [1] or on applying deep learning to generate new DDR1 inhibitors [2]). Notably, KinDEL comprises over 81 million compounds, significantly exceeding the size of typical activity datasets, and contains multiple targets opening new avenues for benchmarking, e.g., multitask learning methods. To further address the raised concerns, **we have expanded our dataset to include one more non-kinase target, Bovine Carbonic Anhydrase (BCA)**. This additional target not only enhances the diversity of our dataset but also offers new possibilities for testing model generalization. The experimental details and correlation plots between replicates of the BCA experiment have been added to the appendix, and the data is already available here: https://kin-del-2024.s3.us-west-2.amazonaws.com/data/bca_1M.parquet 

**2. The benchmark could be improved by providing 3D data.**

Initially, we have not included any 3D data due to the computational cost of molecular docking experiments, which are only an approximation of molecule binding to rigid protein structures. However, after reconsidering the comments regarding the inclusion of 3D data, we decided that KinDEL could benefit from providing docked poses, which can be used to benchmark 3D models on a standardized set of poses. This way the results are not dependent on the docking procedure used by the authors of various methods. Therefore, to facilitate structure-based modeling, **we share 4.2M docked poses for the top 200k DDR1 and MAPK14 hits by target enrichment** from the recommended training sets:
https://kin-del-2024.s3.us-west-2.amazonaws.com/data/poses/2024-11-17_kindel-poses.sdf.gz

We plan to provide docked poses for the entire 81M library prior to publication. We have described our docking protocol in Appendix F of the revised paper. Thank you for this suggestion.

**3. More experimental results including models and new splits are recommended.**

We appreciate the Reviewers’ comments with ideas on how we can improve our benchmark. We computed a new similarity-based data split (see new Appendix E), and all the current models will be tested on this split before the discussion period ends. Additionally, we are training one more graph-based model, Chemprop, to showcase a better representation of state-of-the-art graph neural networks. We hope these additional experiments will significantly enhance the utility and value of our benchmark.

**[1]** Röhm, Sandra, et al. "Fast iterative synthetic approach toward identification of novel highly selective p38 MAP kinase inhibitors." *Journal of medicinal chemistry* 62.23 (2019): 10757-10782.

**[2]** Zhavoronkov, Alex, et al. "Deep learning enables rapid identification of potent DDR1 kinase inhibitors." *Nature biotechnology* 37.9 (2019): 1038-1040.
</content>
</thread_object>

</thread>
<thread>
creation date:
1733268385372

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Thank you for the discussion!

comment:
Dear AC and Reviewers,

We want to thank you for the fruitful discussion during the discussion period. We are glad that all the concerns of Reviewers Z3Ms and 63rq were resolved, and they are recommending the acceptance of our paper. We have also made considerable efforts to respond to the feedback from Reviewers BHqP and 6jAS, though we have not yet received further comments from them following our recent responses and updates. The final revision includes the additional experiments we had run to address their reviews. To facilitate the further discussion between the Reviewers and the AC, we summarize the major changes in the revised paper.

**1. New experiments:** A new cluster-based data split and a more recent model in the benchmark.

We have run experiments on a new cluster-based split. Based on the results, this data splitting method is more difficult for the models than the random split but less difficult than the formerly proposed disynthon split. We have also added a new SOTA graph-based model, Chemprop, which is a more recent model that uses bond message passing. The new results are presented in the extended Tables 1 and 2. The new data split is also visualized in Figure 5a. The details on how the data split was computed and the discussion on the scaffold-based split are presented in Appendix E.

**2. Addition to existing dataset:** Docking poses provided along with the dataset.

High quality docking requires substantial expertise and specialized software in addition to significant computational resources. In order to democratize and standardize access to 3D binding poses for KinDEL, we have provided poses of the ligands docked to DDR1 and MAPK14. This addition makes it possible to evaluate structure-based predictors that use the 3D ligand poses using our benchmark. KinDEL is the first DEL dataset that provides docking poses, which makes this dataset useful for developing structure-based DEL models. The details on the docking procedure as well as examples of the docked compounds are presented in Appendix F.

**3. New target added to dataset:** A new non-kinase target (BCA) included in the dataset.

We have expanded our extensive dataset by introducing data for a new biological target, bovine carbonic anhydrase (BCA). This addition significantly enhances our collection, providing experimental counts in triplicate for all ~81 million compounds, which represents a 50% increase in experimental data. The inclusion of BCA enables researchers to test the generalizability of their models and explore new training methods, such as multi-task learning. Appendices A and B have been updated to detail the experimental procedures and confirm the replicability of BCA results. The data is available in the same S3 bucket as the existing data for the other two targets.

**4. More analysis of data and results:** A more detailed description of the role of the on- and off-DNA testing sets.

In response to the raised comments, we have added more discussion about the on- and off-DNA compounds in the “Held-out Test Set” section on page 4 and information of potential biases due to the DNA strand in the “Challenges and Future Directions” section on page 9. More discussion can be found in our responses to Reviewers Z3Ms and 6jAS.

---

In summary, KinDEL stands out as a comprehensive DEL dataset encompassing approximately 81 million compounds, evaluated against 3 targets, each with 3 replicates. This makes it significantly larger than other chemistry-related datasets, including other DEL datasets, with only Belka published concurrently being of slightly larger size. Our dataset provides full molecules as well as all three synthons used to build the library, offering unique opportunities for modeling applications. Moreover, KinDEL is the first DEL dataset to include docking poses, greatly enhancing its utility. We believe this dataset and its benchmark will serve as a valuable resource for the ICLR audience and the broader machine learning community.


Thank you again,

Authors
</content>
</thread_object>

</thread>


Metareview:
metareview:
This work aims to present a large publicly accessible DNA-encoded library (DEL) datasets, referred to as KinDEL, which comprises two kinases: namely, Mitogen-Activated Protein Kinase 14 (MAPK14) and Discoidin Domain Receptor Tyrosine Kinase 1 (DDR1).
All reviewers recognize the value of such dataset, which may serve as valuable resources for advancing data-driven research in relevant areas.
The provided datasets and benchmarks are comprehensive and the construction process is also well thought-out and reasonable, building on the authors' comprehensive review of relevant literature.
However, the datasets are narrowly focused and not all reviewers are strongly convinced that the constructed datasets/benchmarks will be of interest to the broad ICLR community.
Performance assessment and analysis based on the latest SOTA methods are also insufficient and the paper also does not provide deeper insights regarding the observed evaluation results and the underlying factors of the respective models that led to the results.