dataesr/scientific-paragraphs-categorization huggingface dataset:
Feature selection After preprocessing, feature selection is the next important step for microbiome data analysis in order to identify the most informative genes, taxa or functions. In principle, feature selection is the process of selecting a subset of features from a larger set of available features that are most important in predicting the outcome variable. The goal is to reduce the number of input features required to achieve good model performance, thereby improving the efficiency and interpretability of the model. This also helps to avoid overfitting, a common problem when analyzing high dimensional datasets where the model becomes too complex and starts to memorize the training data instead of learning general patterns. Feature selection techniques can be broadly categorized into three types: filter methods, wrapper methods, and embedded methods.
, 2022). As a result, the estimate is conservative, which is better than being overly optimistic. Essentially, during performance estimation we are not evaluating a specific model instance, but the entire ML pipeline that produces the final model. Lastly, a typical methodological problem in predictive modeling is that of data leakage which can lead to optimistic or entirely invalid models. Data leakage occurs when performing data preprocessing or feature selection on the whole dataset before applying cross-validation. For example, when standardizing the data using the mean and standard deviation of the entire dataset, the rescaling process gains knowledge of the full data distribution, introducing bias on the rescaled values that can affect the performance of the algorithms on the cross-validation test sets.
g. , SMOTE  (Chawla et al. , 2002). General concern with oversampling is the increase of likelihood of overfitting due to exact or synthetic copies of the existing data  (Fernández et al. , 2018). Undersampling methods include random undersampling, where instances from the majority class are randomly removed, and methods that involve heuristics or learning models trying to find redundant examples for deletion or useful examples for non-deletion. However, removing too many samples from the majority class can be a problem, especially if the dataset is small. Oversampling and undersampling techniques can potentially enhance model performance when applied either as preprocessing steps or as integral components of the model itself  (Mihajlović et al. , 2021). On the other hand, class weighting regards the assignment of weights to the classes to balance their contributions to the loss function during training.
By assigning higher weights to the minority class, the algorithm can redistribute its capacity to focus more on correctly predicting the minority class, thus improving the overall performance on the imbalanced dataset. However, finding the right weights can be quite challenging. This strategy, also known as costsensitive learning strategy encourages the model to focus on correctly predicting the minority class, as misclassifying instances of this class incurs a higher cost  (Ling and Sheng, 2010). Regardless of how class imbalance is approached when evaluating the performance of a model on an imbalanced dataset, it is important to use appropriate performance metrics that consider the imbalance (e. g. , balanced accuracy; averaged versions of precision, recall, F1-score etc. ).
, 2004). In contrast, AUPR focuses on the precision-recall trade-off and is particularly useful in imbalanced datasets, when the positive class is of greater interest. It denotes the probability of correct detection of positive instances  (Saito and Rehmsmeier, 2015). Thresholddependent measures, on the other hand, assess classifier performance at a specific decision threshold between 0 and 1. Threshold tuning does not change the classifier quality but can improve the performance metric, while also being one of the simplest approaches to handle a severe class imbalance  (Fernández et al. , 2018 ). Accuracy, for example, calculates the proportion of correctly classified instances over the total number of instances. However, it can be misleading under class imbalance, as it may achieve a high accuracy score by simply predicting that all observations belong to the majority class  (Akosa, 2017).
, 2018)  among others  (Ding et al. , 2014). The nested cross validation involves a double loop procedure, where an inner cross-validation loop is run over the training data and is used for hyperparameter tuning, and an outer one for estimating the performance. Although nested cross-validation is very useful when the dataset is small and the number of hyperparameters is large it is computationally very expensive. The Tibshirani and Tibshirani method does not employ a separate hold out set. Rather it employs traditional K-fold cross-validation estimates to calculate the bias and subtract it from the performance estimates. A similar, but computationally more efficient method that has smaller variance and bias, is the BBC-CV method that was recently presented  (Tsamardinos et al. , 2018). Here, in order to calculate the bias, bootstrap resampling is employed on the pooled out-of-sample estimates collected during cross-validation of multiple pipelines.
Description of the data This dataset  (Barbet et al. , 2023)  gathers 2090 human stool samples characterized by shotgun metagenomic sequencing from 13 public cohorts spanning nine countries (Table  3 ). This data provides the gut microbiota composition in healthy controls and patients with adenoma or CRC. Data were prepared as follows. Sequencing data was downloaded from the European Nucleotide Archive. Reads were quality trimmed and filtered from sequencing adapters using fastp. Remaining contamination by the host genome was filtered out by mapping reads against the human reference genome (T2T-CHM13v2. 0) with bowtie2. Microbial species identification and quantification was estimated according to both human gut reference gene catalog (IGC2, 10. 4 M genes,  Wen et al. , 2017)  and human oral gene catalog (8.
A variable threshold of prevalence across samples in [0-0. 5], with 0. 05 steps was applied to remove features with low prevalence. Figure  2  shows the sensitivity and specificity results for the two best performing models: GBM and RF. We observed that a small filtering slightly improved the performances both on accuracy and computing resources criteria. However, it is noteworthy that no filtering on prevalence at all is also a valid option in terms of performances. As expected, strong filtering on prevalence (>0. 15-0. 2) decreases the sensitivity for GBM and the specificity for RF. Additional analyses of other microbiome datasets (Supplementary material 1 and Supplementary Figure  S1 ), showed that performance was not affected by 0. 2 prevalence filter with regard to 0 prevalence filter in RF models.
However, other models such as PLS-DA got better classification error rates when 0. 2 prevalence filter was applied. The results from these additional datasets indicate that the effect on performance of the low-abundance filter depends both on the ML model applied and on the characteristics of each dataset, being the level of sparsity of the database a key factor to consider. All in all, this fact highlights the importance of including the low-abundance filter as another hyperparameter to tune while training the model by cross-validation strategies. Figure  3  shows the sensitivity and specificity results for all the models with or without the CLR logratio transformation before the modeling process. We observed that for the majority of the models, the CLR transformation decreased the sensitivity of the models, and it was particularly striking for the Glmnet and Glmboost models.
SES, on the other hand, drawing inspiration from causal modeling theory, demonstrates better performance at low sample sizes and leads to smaller feature subsets at the expense of predictive performance. SES also produces multiple signatures that exhibit statistically indistinguishable predictive performances. For modeling, we employed well known linear/interpretable algorithms such as ridge Sensitivity and specificity of the two best performing ML models (GBM and RF) on 100 data split repetitions applied on the CRC dataset with a range of filter on prevalence (shades of blue) or no filter on prevalence (gray).
Indeed, when we employed JADBio on the complete, all-cohorts, dataset we observed that the cohort ID/country exhibited predictive power, indicating inherent variations in gut species between different countries (Figure  4A ). Furthermore, we found that the timepoint of measurement, the instrument model, and the westernization status of samples also possessed predictive value, pointing toward the existence of batch effects (the full description of the metadata features used in the models are provided in Supplementary Table  S1 ). To address these variabilities, we conducted a series of subsequent analyses by splitting the data into the different cohorts. Feature selection identified that the timepoint feature was predictive in the Japanese cohort, the instrument model in the German cohort and the westernization status in the Indian cohort (Table  5 ).
Therefore, we repeated the analysis on the entire sample set after excluding the problematic samples coming from these cohorts. This time, the revised data we divided into two parts: one for training and the other for testing. The revised findings indicated the absence of latent variabilities, suggesting that our modifications successfully controlled for the previously observed effects. The best performing pipeline on the revised data was a combination of SES and random forests, consistent with the majority of separate cohort analyses. A total of 596 different pipelines were evaluated by a repeated 10-fold CV approach (see Supplementary report for details). As shown in Figure  4B , pipelines incorporating SES for feature selection demonstrated higher average performance during training than those with LASSO.
The detailed taxonomy of the species involved is provided in Supplementary Table  S3. Variability in performance was also highlighted by  Wirbel et al. (2019)  where only the predictive performance on several cohorts was examined. This suggests the need for further investigation into the specificity of these microbiome signatures. Interestingly, however, 20 species present in the revised dataset's signature were also found in the signatures generated when analyzing each cohort independently, indicating their potential importance across diverse geographic communities. Among the selected species in the revised dataset's signature, their relevance is in agreement with previous reports in the literature regarding their predictive role in CRC. In particular, considering the top five most important species identified for the revised dataset, excluding gender (Figure  5C ), Fusobacterium gonidiaformans (msp_1081) was detected in colorectal carcinoma relative to normal colon  (Castellarin et al.
Acknowledgments We greatly thank  Emmanuelle Le Chatelier  and  Pauline Barbet  ( Université Paris-Saclay, INRAE, MetaGenoPolis , 78350,  Jouy-en-Josas, France ) for preparing the shotgun CRC benchmark dataset. We also thank  Michelangelo Ceci  ( Department of Computer Science, University of Bari Aldo Moro, Bari, Italy ) and  Christian Jansen  ( Institute of Science and Technology, Austria ) for their interim leadership of the Working Group 3 of the  COST Action ML4Microbiome.
Funding The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was based upon work from  COST Action ML4Microbiome "Statistical and machine learning techniques in human microbiome studies " ( CA18131 ), supported by  COST (European Cooperation in Science and Technology , www. cost. eu). MB acknowledged support through the  Metagenopolis  grant  ANR-11-DPBS-0001. ML acknowledged support by  FCT -Fundação para a Ciência e a Tecnologia , I. P. , with references  UIDB/00297/2020  and  UIDP/00297/2020  ( NOVA Math ),  UIDB/00667/2020  and  UIDP/00667/2020  ( UNIDEMI ), and  CEECINST/00042/2021.
Currently, at least 18 PROTACS are undergoing phase 1/2 clinical trials  25. A compilation of bioactive PROTAC molecules, along with their description in terms of physicochemical properties was recently published and is expected to help campaigns of PROTAC drug design  26. While more and more experimentally determined structures for E3 Ligase-PROTAC-Target systems are being deposited in the Protein Data Bank (PDB)  27  , the data is still sparse for PROTAC design approaches. Thus, computational methodologies are being explored to predict the structure of ternary complexes and, using machine learning methods, degradation rates  [28] [29] [30] [31] [32]. One of the most popular tools for PPI prediction is protein-protein docking, where the prediction depends on two critical steps: sampling of the conformational landscape of the protein-protein complex and scoring of the near-native poses obtained  33.
Our approach is tested on a manually curated dataset of 13 ternary complexes extracted from the PDB and relies on the predictions provided by LightDock  38, 40, 50. LightDock is combined with energy rescoring afforded by VoroMQA  51  and a filtering step based on the minimal solvent-accessible surface distance (SASD) between anchor atoms of the ligands bound to the ligase and target proteins, computed using Jwalk  52. We show that restrained-LightDock simulations produce reasonable PPI structures, which are compatible with experimental structure of the ternary complex, for nearly all systems under scrutiny. We further compare the performance of our method with AlphaFold2-Multimer  46, 53  , highlighting critical limitations in the machine-learning based method for this kind of application, and benchmark its computational efficiency against PROTAC-Model 31 highlighting that the present approach is able to efficiently generate reasonable starting ligase-target binding modes.
Dataset selection and curation Thirteen PROTAC ternary complex crystal structures were extracted from the PDB  27  , selected based on three criteria: (1) that the crystal structure contained an E3 ligase, a ligand molecule, and a target; (2) that the crystal structure resolution was below 4 Å and (3) that the ligand was a PROTAC and not a molecular glue. The dataset is composed of three types of E3 ligases (9 VHL  17, [54] [55] [56] [57] [58] [59]  , 2 Cereblon 60-63 and 2 cIAP  64  ), multiple different PROTAC molecules and a diverse set of protein targets, from kinases to bromodomains or proteins implicated in DNA repair (as WRD5). For the realistic docking case scenario, unbound monomers were selected to be different from the initial 13 crystal structures in terms of interface side-chain packing  54,64- 71.
The SASD filter was calibrated using the predicted PPIs, exploring several SASD ranges (Table  S6 ) and the reference structures in the benchmark dataset with the aim of recovering as many acceptable solutions as possible while also encompassing at least 90% of the SASDs computed on the crystal structures (see Table  S1 ). The rescoring step corresponds to a re-ranking of the LightDock-generated structures in ascending order of the calculated VoroMQA energies. The filtering step shifts all solutions which do not pass the criteria to the bottom of the ranking, maintaining the order obtained from the energy rescoring. This filtering allows one to discard predicted PPI structures where the either or both of the binding sites are occluded, effectively acting as an implicit "PROTACability filter".
The docking workflow recovers PROTAC-mediated ternary complex-compatible PPIs To evaluate the accuracy of our protocol, we first carried out redocking experiments on the benchmark dataset of 13 PROTAC-mediated complexes (Table  S1 ). Here, we consider that the docking was successful if it generated at least one acceptable or higher quality solution within a given cohort (Top 1, 5, 10, 20, 50 or 100 ranked poses). From the information gathered in the literature, in particular the reports about ternary complex flexibility from Dixon and co-workers  45  , we hypothesized that accurate prediction of these interfaces would be a challenge for an unbiased docking algorithm. A way to help drive the docking towards the correct interface is through the introduction of contact residue restraints, which we pursued as detailed in the experimental section.
As shown in Figure  3A , the filters improved remarkably the performance of the workflow, with the accuracy rising from 46% to 77% (Top 20, 6 Å contact residue restraints). However, the enhancement in performance observed here is still a product of a dual effect: the docking algorithms' ability to produce meaningful solutions and the accuracy of the post-processing procedure, i. e. the energy-rescoring and SASD filtering. To ascertain the performance of the post-processing approach, we removed those systems for which no meaningful docking solutions were found for the best performing setup, which employs 6 Å contact residue restraints (Figure  3B ), two VHL cases (PDB IDs 7JTP, 7KHH, Figure  S4 ). In this analysis, we considered successes the cases where LightDock was able to generate at least one acceptable solution for that particular system, a classification which allows to decouple the docking engines' ability to produce meaningful solutions and the post-processing filters' ability to recover those solutions.
Discussion In this study, we propose a novel workflow for the generation of PROTAC-compatible PPI structures in the case that no known PROTAC molecule exists, no information of a possible linker is available, and the ligase-recruiter and target-warhead co-crystals are available. To this end, we manually curated a dataset of 13 PROTAC-containing ternary complexes from the PDB to serve as a benchmark dataset. Using this dataset for protocol benchmarking, we carried out both redocking and realistic docking experiments (see Results) where the accuracy of the docking was measured by evaluating the quality of the predicted structures using CAPRI and DockQ criteria  [79] [80] [81]. We found that both the inclusion of contact residue restraints and the rescoring using the energy function of VoroMQA 51 , were fundamental to increase the accuracy of the calculations in both redocking and realistic docking experiments.
However, it requires a priori knowledge of the PROTAC to use, which is critical for the filtering and retrieval of the protein-protein docking poses closest to the reference structure. We hypothesize that this is not the most optimal approach, since according to the data from Dixon and co-workers  45  , the crystal structure is one possible solution but it is not necessarily the only acceptable solution and trying to reproduce it may lead to discarding other biologically relevant conformations. Nonetheless, we compared the computational cost and the accuracy of our method to PROTAC-Model by Weng et al  41  (Table  S5 ) across the systems shared between the two studies without employing PROTAC-Models' Rosetta refinement, using 20 cores. From this comparison, we observe that while PROTAC-Model is superior to our approach at reproducing VHL-based ternary complex crystallographic structures, our method was (1) able to recover solutions for both cIAP systems and (2) performs slightly better than PROTAC-Model for the Cereblon systems studied.
Conclusion In this study we have described a new LightDock-based protocol for PROTAC drug design. Our workflow combines restraint-enabled LightDock simulations with energy-based rescoring from VoroMQA and SASD calculations from Jwalk to yield at least acceptable quality PPI complex structures, starting only from the ligand-bound monomeric form of the E3 ligase and the protein target. The protocol is generalizable to any E3 ligase/target pair, fast and accurate, achieving 92% and 77% accuracy for the top 20 cohort on the redocking and realistic docking experiments, respectively, when benchmarked against a manually curated and diverse dataset of PROTAC-mediated ternary complexes. In the case that no PROTAC molecule is known to bind the ligase/target pair of interest, our approach is able to provide initial configurations which reproduce the protein-protein binding interface and provide the minimal solvent accessible path connecting the ligand and target ligands, potentially accelerating PROTAC drug design campaigns by narrowing down the chemical space to be searched for linker selection.
Acknowledgements Computer hours awarded to the  GENCI  project number  2022-A0120713456  on the  Jean-Zay  and  Joliot-Curie  Rome clusters of the  French National Supercomputing Center  are gratefully acknowledged. We gratefully acknowledge CC-IN2P3 (https://cc. in2p3. fr) for providing a significant amount of the computing resources and services needed for this work. The authors also acknowledge the support provided by the  CNRS  and  PharmCADD.
Financial support Brian Jiménez-García  (BJG) is employed by  Zymvol Biomodeling  on a project which received funding from the  European Union 's  Horizon 2020 research and innovation programme  under  Marie Skłodowska-Curie  grant agreement No. 801342  (Tecniospring INDUSTRY) and the  Government of Catalonia's Agency for Business Competitiveness (ACCIÓ). PCTS, JM, GPP, GL and RP are supported by the  French National Center for Scientific Research (CNRS). Further funding of GPP and PCTS came from a research collaboration with  PharmCADD.
A di-PWM and score computation of a word is illustrated in Fig. 1a. For instance, the HOCOMOCO database (v11) stores di-PWMs of binding motifs of Human and Mouse transcription factors (292 and 257, respectively), which were directly computed from experimental ChIP-Seq data  (Kulakovskiy et al. 2018). For detecting new binding sites, it was shown that di-PWMs provide enhance sensitivity compared to classical PWMs  (Kulakovskiy et al. 2013). di-PWMs also obtained successful results in the DREAM-ENCODE challenge from 2017. To our knowledge, only SPRy-SARUS and MOODS can find occurrences of di-PWMs in long sequences. SPRy-SARUS is an efficient standalone Java program to search for di-PWMs, which is coupled with MoLoTool, a web tool allowing visual inspection of occurrences in short sequences  (Kulakovskiy et al.
2018). MOODS, a tool to search for PSSM/PWM, can also handle di-PWMs  (Korhonen et al. 2017 ) (v3 available as Cþþ code and Python package). Both adopt a window scanning strategy, while dipwmsearch uses an enumeration strategy. Hence, we provide a Python package for searching di-PWM in sequences: it can be easily installed via conda and offers several functions that can be used in Python programs. The user can control the.
Results We compared the core enumeration algorithm to SPRy-SARUS in terms of efficiency by searching each Human di-PWM from HOCOMOCO on Human chromosomes 3 and 15 with four different score ratio thresholds (0. 8, 0. 85, 0. 90, and 0. 95). The score ratio threshold is a way to set a relative score threshold that is comparable between different di-PWMs (it was developed for PWMs-see the FAQ of JASPAR database). The score threshold h is computed as follows: h ¼ ðscore max À score min Þ Ã ratio þ score min , where score min and score max are respectively the minimum and maximum achievable scores for the input di-PWM. Note that dipwmsearch provides an alternative to the ratio threshold: the user can give a P-value threshold to limit the score of occurrences found. First, the results shown in Fig.