dataesr/scientific-paragraphs-categorization huggingface dataset:
csh script in SIFT 5. 1. 1 59. The siftr R package (https://github. com/omarwagih/siftr) was used to generate SIFT scores with parameters ic_thresh = 3. 25 and residue_thresh = 2. PolyPhen2 scores were obtained from http://genetics. bwh. harvard. edu/pph2/bgi. shtml using the batch query tool on 6 August 2018. The GRCh37/hg19 genome assembly and the HumDiv classifier model were used for considering missense annotations against canonical transcripts only. Allele frequencies. A total of 3,198,692 coding variants in H. sapiens for over 65,000 individuals was collected from the ExAC Consortium 29 in the ANNO-VAR 60 output format along with corresponding adjusted allele frequencies. Ensembl transcript positions were mapped to UniProt by performing Needleman-Wunsch global alignment of translated Ensembl transcript sequences against the UniProt sequence using the pairwiseAlignment function in the Biostrings R package.
Acknowledgements The IntAct database and EMBL-EBI-based authors received funding from  EMBL core funding and Open Targets  (grant agreement  OTAR-044 ). The DIP database is funded by  NIH  grant  R01GM123126. MINT received support from  ERC  grant ' DEPTH  project of the  European Research Council  (grant agreement  322749 )'. The  British Heart Foundation-University College of London (BHF-UCL ) curation team is funded with the  British Heart Foundation  grant  RG/13/5/30112. UniProt curation activities at  EMBL-EBI  and the  Swiss Institute of Bioinformatics  are funded by the  National Eye Institute, National Human Genome Research Institute, National Heart, Lung, and Blood Institute, National Institute of Allergy and Infectious Diseases, National Institute of Diabetes and Digestive and Kidney Diseases, National Institute of General Medical Sciences , and  National Institute of Mental Health of the National Institutes of Health  under Award Number [U24HG007822],  National Human Genome Research Institute  under  Award Numbers  [ U41HG007822  and  U41HG002273 ], and the  National Institute of General Medical Sciences  under  Award Numbers  [ R01GM080646 ,  P20GM103446  and  U01GM120953 ] (the content is solely the responsibility of the authors and does not necessarily represent the official views of the  National Institutes of Health) ;  Swiss Federal Government  through the  State Secretariat for Education, Research and Innovation ; and aforementioned  British Heart Foundation  grants and EMBL core funding.
DisGeNET is supported with  EU - FP7  funds from  ISCIII-FEDER  ( CP10/00524 ,  CPII16/00026 ),  IMI-JU  (grant agreement no. 116030 , TransQST) and EFPIA companies in kind contribution, and the  EU   H2020 Programme 2014-2020  (grant agreements no. 634143 ,  MedBioinformatics  and no. 676559 ,  Elixir-Excelerate ). The  Research Programme on Biomedical Informatics (GRIB)  is a member of the  Spanish National Bioinformatics Institute (INB) ,  PRB2-ISCIII  and is supported by grant  PT13/0001/0023 , of the PE I+D+i 2013-2016, funded by  ISCIII  and  FEDER. The  DCEXS is a 'Unidad de Excelencia María de Maeztu' , funded by the  MINECO  (ref:  MDM-2014-0370 ). I. J. and group supported in part by  Krembil Foundation ,  Ontario Research Fund  (# 34876 ), and  Canada Foundation for Innovation  (CFI # 225404 , # 30865 , # 33536 ).
We find that ligand-based features have lower predictive power (r P ¼ 0. 69, R 2 ¼ 0. 47) than structure-based features (r P ¼ 0. 78, R 2 ¼ 0. 60). Their combination maintains high accuracy (r P ¼ 0. 73, R 2 ¼ 0. 50) on the internal test set, but it shows superior robustness on external datasets. Further improvement and extending the training dataset to include xenobiotics, leads to a novel high-throughput affinity prediction method for ERa ligands (r P ¼ 0. 85, R 2 ¼ 0. 71). The presented prediction tool is provided to the community as a dedicated satellite of the @TOME server in which one can upload a ligand dataset in mol2 format and get ligand docked and affinity predicted.
The parallel approach compares the selected compounds of both methods and retrieves either a consensus (selected by both) or a complementary selection (top molecules from each approach)  (Lavecchia and Di Giovanni, 2013). Alternatively, one might apply a weak similarity restraint such as a molecular shape restraint for the ligand (to be classified as a shape-matching LBVS method) during the docking process in SBVS as it is implemented in the docking software PLANTS  (Korb et al. , 2009). In the present study, we take advantage of a new interface between PLANTS and the web server @TOME  (Pons and Labesse, 2009)  to screen multiple conformations in parallel (to be described in more details elsewhere). It also allows us to systematically deduce a shape restraint and binding site boundaries based on the geometry of the original ligand from each crystal structure in a fully automatic manner.
BindingDB dataset Two sets of experimentally tested ligands for the human ERa (UniProtID: P03372) were extracted from BindingDB  (Gilson et al. , 2016; Liu et al. , 2007 Liu et al. , ) (2018 Liu et al. , dataset, updated 2018-04-01)-04-01). One set contains ligands with known inhibitory constant (Ki) as affinity measure, and a second set contains ligands with half maximal inhibitory concentration (IC50) as an affinity proxy. A few peptides and a series of boron cluster containing molecules were removed from both datasets, as it was not possible to generate proper 3D conformations or charges for these molecules. The final sets contained 281 ligands (Ki set) and 1641 ligands (IC50 set), respectively, with an overlap of 48 common compounds. Overall, both datasets show a similar compound diversity (compare Supplementary Fig.
In-house xenobiotic dataset The xenobiotic chemical data that was used first as an external testing dataset and afterwards to build an extended training set, is an in-house dataset of 66 ligands with measured affinities for ERa  (Grimaldi et al. , 2015). These extra compounds correspond mostly to bisphenols, halogenated compounds, as well as phytoestrogens (natural fused heterocycles and macrocycles partially micmicking estradiol).
FDA ERa dataset In order to have a second external validation, we used the Estrogen Receptor targeted dataset from the Endocrine Disruptor Knowledge Base (EDKB) provided by the U. S. Food & Drug Administration (FDA), named here ER-EDKB dataset. The dataset contains 130 ER binders and 101 non-ER binders including natural ligands and xenochemicals that are structurally different from drug-like molecules. For ER binders, the binding affinity measure is reported as a relative binding activity (RBA), which is based on an assay using rat uteri. Those cell-based measurements are influenced by different factors, such as cellular permeability, and are unfortunately not directly comparable with direct Ki measures. Nevertheless, we predicted affinities using all models and transformed the measured RBA values back to pIC50 values (pIC50 ¼ log 10 ðRBAÞ À 8).
The automatic virtual screening procedure implemented in the @TOME server uses the docking program PLANTS with its shape restraint functionality (with a weighting of -3, the default value suggested by the software manual), using the original ligand of the screened structure as a pharmacophore. Of note, this ligand is also used to define the   and Frog2 ) were employed to generate a total of five ligand sets. Those were submitted to the @TOME server for docking and complex evaluation. The @TOME output datasets 'MMFF', 'Gast', 'BDB', 'OB3D' and 'Frog3D' (containing the results of 20 dockings per ligand in different structures) were grouped in three combined datasets, a different charge dataset 'dCharge', a different conformation dataset 'dConf', and an 'ALL' dataset boundaries of the binding site to be screened (using a distance cutoff of 8 A ˚).
Ligand molecular descriptors In order to include more information about the small molecules being screened, molecular descriptors were calculated using the Chemistry Development Kit (CDK)  (Willighagen et al. , 2017) , a collection of open source Java libraries for chemoinformatics, through its R interface rcdk  (Guha, 2007). The descriptors were selected based on their ability to represent the diversity of the ligand dataset, taking into account their orthogonality, and based on their variable importance score during model training. The final set of 11 QSAR molecular descriptors includes topological, geometrical, constitutional and charge based descriptors and is listed in Table  2  with CDK descriptor name, the used abbreviation and a short description.
Combined structure/ligand descriptors All 5 docking datasets (originating from the 5 different ligand sets) provided 19 structure-based docking metrics for the 20 docking poses computed for each ligand. For each metric, median and standard deviation were computed and used as a unified instance. Ligandbased variables (11 CDK molecular descriptors) were added to the 19 structure-based metrics. A correlation matrix with all descriptors used for the Ki-BDB dataset is provided as heatmap (see Supplementary Fig. S5 ). Alternatively, the commonly used MACCS fingerprints (166 features)  (Durant et al. , 2002; Taylor, 2007)  were also tested for comparison.
Algorithm selection and training For all analyses, calculations and machine learning, the R language (version 3. 2. 4) was used with RStudio (employed packages are listed in Supplementary Table  S1 ). In an initial test on the BDB Ki dataset we assessed the performance of 7 machine learning algorithms (see Supplementary Fig. S3 ): Random Forest (RF), Gradient Boosting Machine (GBM), support vector machine (SVM) with a radial kernel (SVM_R), a polynomial kernel (SVM_P) and a linear kernel (SVM_L), linear regression (LinReg), decision tree (CARTree) and Partial Least Squares (PLS). All algorithms were employed with default variable settings with the R package 'caret'. In order to avoid over-fitting of the models, we used stratified 10-fold cross validation repeated 10 times for all models in this study (unless otherwise indicated).
Comparison of different tree-based algorithms The RF algorithm we used, has only one tunable hyperparameter that can be adjusted for the present dataset. Therefore, we wondered whether other tree-based ensemble algorithms with more tunable hyperparameters offer an improved prediction accuracy when tuned more carefully. In total, five different tree-based algorithms were employed on the same Ki BindingDB2018 dataset for affinity prediction and subsequent performance comparison. They are: random forest (RF), regularized random forest (rRF), global regularized random forest (rRFglobal), Extreme Gradient Boosted Trees (xgbTree) and Extreme Gradient Boosted Trees with dropout (xgbDART). Here, Bayesian optimization was employed to select the best hyperparameters (5 to 7 depending on the method), which demands a substantially increase in computational expense compared to the one-variable optimization required for the RF algorithm.
Random Forest regression modeling Random forest models were trained on each dataset separately ('MMFF', 'Gast', 'BDB', 'OB3D' and 'Frog3D'), on the combination of the three different 3D conformation datasets ({'BDB', 'OB3D', 'Frog3D'} ¼ 'dConf'), on the combination of the three different partial charge datasets ({'MMFF', 'Gast', 'BDB'} ¼ 'dCharge') and on all five datasets combined ('ALL') (compare Fig. 1 ). Besides the Pearson correlation (r P ), two further regression evaluation metrics were used to evaluate the model performance on the external test set. First, the coefficient of determination (R 2 ) is calculated using the sum of squares method. The second metric, the Root Mean Square Error (RMSE) is the average deviation of the predictions (predicted affinities) from the observations (measured affinities).
Predictions using re-scoring methods In a first attempt, the predictive power of the four different scoring functions implemented in the @TOME server was assessed. The Pearson correlations between affinity measurement and the median scores (calculated on 20 docking poses per ligand) are very low for all generated datasets (see Table  3 ). Even the most recent scoring functions (MedusaScore and DSX) performed poorly in this test. Interestingly, the selection of the best pose among the 20 computed ones slightly improves the correlation between predicted and measured affinities for 3 scoring functions but for MedusaScore which appeared as the most robust and the best for the various ligand description schemes. However, the overall correlation is too low for fine ligand affinity prediction and indicates a limitation of the general-purpose scoring functions, but the better Spearman correlation (see Supplementary Table  S2 ) suggested a better ranking that could be useful to guide machine learning.
Structure-based and ligand-based partial models To investigate the actual affinity prediction capabilities of structurebased and ligand-based variables, partial models were trained using the 19 structure-based metrics or the 11 ligands-based metrics on the same dataset named BDB. Affinity predictions made on the heldout 20% test set are shown in Supplementary Figure  S6. The docking-metrics only model (r P ¼ 0. 78, r S ¼ 0. 77 and R 2 ¼ 0. 60) outperforms the molecular-descriptors only model (r P ¼ 0. 69, r S ¼ 0. 70 and R 2 ¼ 0. 47). Interestingly, the combined model (trained on BDB using simultaneously Vconf and Vcharge data) has Pearson correlation coefficient, Spearman's rank and R 2 value in between the reduced-variable models (r P ¼ 0. 73, r S ¼ 0. 74 and R 2 ¼ 0. 50).
Random Forest model trained on MACCS fingerprints In this context, it might be interesting to add more information regarding the chemical nature of the ligands studied. Instead of using a reduced set of ligand-based parameters, we turned to use a more The boosted tree models xgbTree and xgbDART appeared to outperform slightly the RF model on this 'ALL' dataset. But the reverse was true when evaluating the corresponding models onto the FDA dataset (see below). Most of the differences are weak and may not be significant. Accordingly, the more complex implementations did not provide significant increase in performance and they were not studied further.
Random Forest regression-model testing Most remarkable is the strong increase in accuracy when using either the 'dConf' model trained on the three different 3D conformation datasets ('BDB', 'OB3D' and 'Frog3D') or the model trained on the fully combined 'ALL' dataset comprising all five datasets ('MMFF', 'Gast', 'BDB', 'OB3D' and 'Frog3D') (compare also Supplementary Fig. S7 ). Interestingly, using different charge models improves affinity predictions, but slightly less efficiently than using different 3D conformations. This is probably due to the fact that the binding pocket of ERa is mostly hydrophobic and therefore the ligands show the same property and partial charges are predominantly found to differ only marginally.
Analysis of variable importance To assess the impact of the various parameters from structure-based and ligand-based scoring functions, the variable importance was tracked during training of the RF models. The 30 most important variables for the models trained on the 'ALL' dataset is shown in Figure  2. Overall, all models have a rather similar variable importance profile (data not shown). Noteworthy, the most important variable 'Tanimoto_Med' is the same for all trained models showing its outstanding importance. It represents the median Tanimoto score calculated between the docked ligand and the 20 shape restraints (or 'anchors') present in the targeted structure. This may reflect the importance of using structures bound to similar ligand to ensure proper affinity predictions. The second and third most important variables are 'nRotB.
Model evaluation on an in-house xenobiotic dataset We took advantage of a complementary and independent datasetthe xenobiotic chemical data of 66 ERa binders to evaluate the robustness of our models. Our models performed rather poorly on this dataset with a Pearson correlation (r P ) of 0. 48 for the best Random Forest model BDB-Ki (and 0. 40 with the BDB-KiþMACCS model). For the partial models, docking-metrics-only and molecular-descriptors-only, as well as the MACCS-only model, correlations are even lower with r P of 0. 45, 0. 31 and 0. 13, respectively. This underlines the improved robustness of the BDB-Ki model combining SBVS and LBVS features (compare also Supplementary Fig. S8 ). Importantly, the chemical nature of most xenobiotics differs significantly from most of the drug-like compounds from the BDB dataset used for training.
As such, small xenobiotics (including the small bisphenols) occupy only partially the hydrophobic cavity and often also present numerous halogen substitutions (that are notoriously hard to model). Furthermore, for some of the small xenobiotics we cannot rule out the possibility that two molecules may bind simultaneously (with synergetic effects). This result prompted us to combine these xenobiotics and BDB Ki dataset into an extended training set to build a new RF model with improved performance (Fig. 3 ).
Model evaluation on FDA ER-EDKB dataset We then evaluated our two best models on a reference dataset comprising both 322 drug-like and xenobiotic compounds (see Table  4  and Supplementary Table  S3 ). At the first glance, the predictions made using the original model (trained on only BDB-Ki) showed a lower performance especially on the edges of the affinity ranges with both overestimated affinities for small and weak binders (e. g. alkylphenol) and underestimated predictions for tight binders such as rigid and compact agonists. Indeed, the BDB dataset is mainly composed of large and high-affinity antagonists. Accordingly, some FDA compounds such as high affinity agonists, appear as strong outliers. Most remarkable is the benefit of adding a complementary dataset of 66 xenobiotic compounds to the initial 281 ligands from BindingDB (see Table  4 ).
Here, the best Pearson correlation (r P ) of 0. 75 is attained with the model trained on 'ALL' datasets including the xenobiotics and the model trained on a single dataset (BDB) including the xenobiotics also shows a high r P of 0. 71. Accordingly, the nature and diversity of the ligands matter, so that, proper coverage of the studied chemical space, in the training dataset compared to the testing one is essential. The model trained on a single dataset without the xenobiotics has already a decreased r P of 0. 58, whereas the partial models, docking-metrics-only and molecular-descriptors-.
Model evaluation on BindingDB-IC50 dataset Finally, the most extended and reliable dataset we used for evaluating our RF models is the BindingDB 2018 IC50 dataset which includes 1641 entities. Interestingly, the model trained on the Ki dataset already performed well against IC50 data suggesting a strong robustness. Training and testing the IC50 dataset (1641 compounds versus 281 for the Ki dataset) also provides some insights into dataset size requirements for the studied target. First, the performance on the IC50 test set (r P ¼ 0. 87) is better than on the Ki dataset (r P ¼ 0. 77) (compare Table  5 ). Then, cross-predictions were computed by either using the model constructed on the Ki dataset for predictions on the IC50 dataset, or employing the model constructed on the IC50 dataset for predicting the Ki dataset.
In that case, it seems that the small Ki test set (56 compounds) does not allow optimal validation as it shows a significant drop in performance compared to the Ki training set (0. 49 versus 0. 64). On the contrary, the Ki-ALL model showed similar performance on both the IC50 training and testing sets (1319 versus 322 entities). We also evaluated our last model trained on the extended dataset including both the Ki dataset and the xenobiotic dataset on the largest available IC50 dataset from BindingDB (compare Table  6 ). Good predictions were observed for the IC50 dataset although the addition of the xenobiotic dataset did not bring any improvement (nor any deterioration) for that particular dataset. For comparison of all trained model see Supplementary Figure  S10. Again, these results suggests that our final model is rather robust.
Conclusion We provide an original in silico method for accurate binding affinity predictions that takes advantage of structural ensembles, of a limited number of structure-based metrics (19) and of ligand-based descriptors (11) in a unique combination. This combination led to a prediction tool outperforming our other models based either on SBVS or on LBVS features when we take into account not only the overall performance on the internal testing set but also the robustness on a range of distinct datasets. This is true also with the use of many more features as exemplified here with the MACCS fingerprints (166 bits). Our work also confirmed the performance of Random Forest over other machine learning approaches as previously noticed  (Russo et al. , 2018). In some cases, higher accuracy was reported but for smaller compound libraries  (Hou et al.
, 2018). Accordingly, our results present one of the largest validation surveys (1641 ligands from the BDB IC50 dataset) and best performing tools for affinity prediction against ERa. As major advantages, RF algorithms handle non-linearities, numerical and categorical variables, and they give estimates of variable importance and generalization error. By training our model in parallel on various types of partial charges and/or 3D builders, we believe our tool will be more versatile and robust to variations in the way the submitted compound libraries are generated. Noteworthy, the user has simply to upload one single dataset to EDMon, where the submitted chemical compounds will automatically be docked and their theoretical affinity for ERa be computed. With this tool, one can easily and rapidly evaluate new compounds either to find putative binders of ERa or to check the absence of binding to this frequent secondary target, in order to avoid potential side-effects.
Pearson correlations (r P ) on all five datasets between experimental affinities and scores from four scoring functions Plants, MedusaScore, DSX and XScore, of (1) the best pose selected by @TOME, and of (2) the median scores of the four scoring functions, calculated on 20 dockings per ligand on all five datasets MACCS. A new random forest model was trained on MACCS fingerprints representing the ligands only, without providing any structural docking data. This resulted in a Pearson correlation (r P ) of 0. 76, Spearman's coefficient (r S ) of 0. 76 and an R 2 of 0. 57 on the Ki test set, midway between the two partial models compared above (molecular descriptors only model and docking metrics only model). Combining MACCS with docking-based features improves the overall performance on the training and left-out testing dataset (r P ¼ 0.
81, r S ¼ 0. 82 and R 2 ¼ 0. 61) but further evaluation using external datasets suggested some overfitting (see below). Dataset name Plants MedusaScore DSX XScore (1) r P on predictions for the best pose Gast 0. 042 0. 154 0. 129 0. 060 MMFF 0. 063 0. 182 0. 157 0. 082 BDB 0. 038 0. 111 0. 118 0. 076 OB3D 0. 109 0. 180 0. 143 0. 129 Frog3D 0. 022 0. 132 0. 118 0. 040 (2) r P on predictions over 20 poses Gast À0. 031 0. 204 0. 019 0. 049 MMFF À0. 025 0. 192 0. 038 0. 054 BDB À0. 019 0. 087 0. 022 0. 059 OB3D À0. 017 0. 175 0. 008 0. 050 Frog3D À0. 048 0. 199 0. 005 0. 036 P ) of 0. 77 (and standard deviation of 0. 014), compared to the models trained on single datasets with a mean r P of 0. 73 (and standard deviation of 0. 029).