dataesr/scientific-paragraphs-categorization huggingface dataset:
CLADE_HHblits and CLADE BEv HHblits use the CLADE approach, but replace PSI-BLAST models and sequence-profile search with HHblits models and profile-profile search. They allow us to verify whether a profile-profile approach combined with multiple models improves CLADE performance. The test considers, one at the time, ASTRAL domain families and uses a leave-one-out cross validation strategy to verify how many times models constructed on all-but-one sequences help to correctly annotate left out sequences. For all tools, results are reported in Table  1 ; it is important to stress that the test is very stringent because it is realized at the SCOP family level. (For the details on the implementation of the test, see section "Experiment on SCOP datasets" in Methods. ) CLADE multi-source strategy clearly outperforms the mono-source strategy as illustrated by HMMScan and HHblits results.
This test allows us also to compare CLADE with SUPERFAMILY  [18] , a probabilistic profiles library representing all structural domains present in the SCOP database  [41]. Note that CLADE has been constructed in the same spirit as SUPERFAMILY: both systems generate multiple models for a given domain, but they select among the predictions of these models with a different computational approach. SUPERFAMILY selects domains by looking at their best E-value, while CLADE exploits best E-values and also agreement among different models (second step) and domain co-existence (third step). Note that CLADE BEv selects among different predictions of the models by using best E-values, as SUPERFAMILY does, and that domain co-existence is not tested by CLADE BEv on the SCOP sequences, since the testing datasets contain just one domain.
They can be fruitfully used by the biologist to annotate sequences. Analysis of datasets of randomly generated sequences. We performed several statistical tests to show that CLADE predicted domains are likely true positives. For this, we estimated False Discovery Rate (FDR) expressing how many times a tool, namely CLADE, HMMScan and HHblits, recognizes a known domain in a reshuffled sequence. We considered the problem of annotating sequences containing a single domain. We used four different random generations of amino-acids sequences and constructed two H0 and two H1 hypotheses. Based on the sets of sequences generated by these statistical hypotheses, we evaluated CLADE, HMMScan and HHblits. The evaluation of HHblits is included whenever the multi-source strategy is directly compared with the mono-source strategy.
The generated sequences are consistent with a sequence family consensus and they are useful to test domain annotation. Next, we run HMMScan and CLADE for annotating domains in these sequences. HHblits is run by constructing profiles for each query sequence with hhblits over HHdb, the HHblits database, and by using HHsearch for a profile-profile comparison. Since the sequences come from specific domain families, we checked whether CLADE, HMMScan and HHblits could predict the right domain family for each sequence. This experiment was repeated 20 times and we evaluated sensitivity, PPV and F-score on the 296 620 generated sequences, or their associated profile. The plots describing CLADE, HMMScan and HHblits sensitivity at different PPV values, and the number of domains predicted by the three tools at different F-scores, are illustrated in Fig 3A  (top  and bottom ).
falciparum sequence. In this respect, it should be noticed that: 1. here the task is somewhat easier, since HMMScan is asked to annotate a domain sequence and not a larger P. falciparum sequence where the position of the domain has to be identified. 2. we constructed CCMs starting from specific sequences from which CLADE annotated the corresponding P. falciparum domain. They are possibly heterogeneous and they are supposed to guide HMMScan domain recognition, since we know from CLADE analysis that their reference sequence can help annotation. Yet, the lower HMMScan performance demonstrates that the combination of several CCMs (multiple-source modeling) is important for annotation. HMMScan does not use the idea of agreement between heterogeneous models while CLADE does. In conclusion, CLADE, HMMScan and HHblits predict more easily domains in the first H1 dataset than in the second one (see plots on the bottom of Fig 3A  and 3B , where the y-axes have not the same scale).
By comparing the plots on the top of Fig 3A  and 3B , we note that CLADE performs similarly to HMMScan and HHblits on the first dataset, while on the second one, constructed from a list of hard examples of remote homology, the difference among the three systems is much more important. The heterogeneity of the domain models appears crucial for obtaining an improved performance (considering clans or not) and a successful remote homology detection. Clearly, HHblits profile-profile comparison improves the mono-source strategy performance of HMMScan, but it remains poor when compared to CLADE multisource strategy. CLADE FDR computed on multi-domain sequences. We estimated CLADE FDR on the prediction of multi-domain proteins. To do so, we consider a FDR strategy that is based on the first H0 hypothesis introduced above, generating random sequences that preserve the same amino-acids composition of the original sequences.
Here, we generated random sequences starting from the 5 542 P. falciparum sequences. For each of these sequences, we concatenated the real sequence to its random reshuffling (Fig  4A ). Then, to annotate the domains and evaluate the performance of the systems with an estimation of their FDR, we run CLADE and HMMScan on this dataset of sequences and HHblits on the associated profiles. Several For each original protein sequence, we make predictions on it and on twenty shuffled sequences concatenated to the original sequence, to allow "real" domains (Y, S) to boost false predictions on the shuffled sequence (domains V, Z, X) when using domain co-occurrence. The estimated FDR is the ratio of false predictions per protein to the total number of predictions per protein. Notice that domain co-occurrence might imply the original protein sequence to be re-annotated (see domain V in the bottom sequence of the drawing).
B. Illustration of the four types of annotation that can be induced by the concatenation with a shuffled sequence discussed in the text. C. The y-axis is the number of predicted domains per protein ("signal"), while the x-axis is the FDR ("noise"), so better performing methods have higher curves (more signal for a given noise threshold). CLADE (red) outperforms HMMScan (black), HHblits (green) and DAMA (pink) on the two datasets, 1-mer (top) and 4-mer (bottom), obtained by randomly reshuffling P. falciparum sequences (see text). CLADE has been tested under several restrictions and the resulting FDR curves have been added to the plot: CLADE ALV (grey), CLADE BEv (blue) and CLADE BEv -no-cut-off (orange). The inset plot zooms the curves on small FDR values (< 0. 001). doi:10. 1371/journal. pcbi.
This factor seems to particularly affect HHblits in 4-mers, sequences that are harder to analyze than 1-mers. Second, the lack of domain specific cut-off acceptance for HHblits, contrary to HMMScan and CLADE that filter out a large number of false positives due to suitable domain specific cut-offs. Note that such cut-offs are crucial for CLADE. In fact, when we test CLADE BEvno-cut-off, that is the version of CLADE that does not include the SVM filter, that does not use domain specific cut-offs and that considers a score system based on best E-values only, the behavior of the system is essentially identical to the one of HHblits on 1-mer as illustrated in Fig 4. Compare it with the curve associated to CLADE BEv also. (See curves on the 4-mer plot as well. ) In conclusion, we started from random sequences, relatively far from real protein sequences, and showed that CLADE has a significantly different behavior from HMMScan and HHblits on these datasets.
Analysis of the dataset of all P. falciparum proteins Upheld by CLADE performance on the SCOP benchmark datasets and the FDR tests, we checked whether CLADE could significantly contribute to the identification of domains within the full set of P. falciparum proteins. A large number (2464) of P. falciparum proteins has no identified domain in PlasmoDB and it remains with no domain identification (2068) even after the analysis of existing in silico predictive methods  [2, 24, 26]. We performed a large scale domain prediction and compared CLADE results against annotations obtained with HMMScan. All evaluations reported below were realized with the same CLADE parameters (with an Evalue cut-off at 1e-3 and by adopting a specific SVM probability cut-off for each domain). See section "CLADE pipeline, parameter settings and tools used in CLADE" in Methods.
Discussion When sequences in a protein domain family are too divergent, signals of homology are easier to trace with CCMs rather than with models based on the consensus of homologous sequences spanning the entire phylogenetic tree (reminiscent ideas were presented in  [7, [18] [19] [20] 45] ). CLADE shows that, by combining in a unique tool CCMs and consensus models, the predictive power can be highly reinforced. In CLADE, CCMs have been constructed for domains in the Pfam database. In this respect, it is worth to highlight that CCMs are basically different from: 1. Pfam models characterizing Pfam clans  [46]  whose purpose is to group different families together, possibly new protein families that appear to have arisen from a single evolutionary origin; 2. Pfam models characterizing Pfam domain families (called SCMs in CLADE).
falciparum sequences are more easily identifiable by CCMs generated by phylogenetically distant species rather than CCMs coming from Alveolata species, other observations strongly reinforce the interest in looking at different clades while identifying domains: 1. multi-domain proteins identified with CCMs originated in different clades, suggest processes of domain evolution that are independent within each protein (see the three domains identified with models constructed from Alveolata, Amaebozoa and Cryptophyta sequences in Fig 8 ); 2. new domain families are periodically added to databases like Pfam and annotation becomes gradually more precise as a function of this addition (see the impact of the Pfam 27 enrichment on the Chlamydomonas reinhardtii protein architecture compared to Pfam 24 in Fig 7A  and 7C ); 3.
5. The pool of approximately 350 CCMs associated to each domain could be extended to provide an enriched annotation. Several databases, such as SCOP, CATH, Gene3D, propose domains that are not shared with Pfam. From these domains new models can be constructed and added to the CLADE model library. This perspective demands the development of criteria to avoid redundancy in the model library and to ensure a reasonable computational time. These methodological improvements are general, independent from specific genome characteristics, and guarantee the strategy to be applied to any genome. They will likely be able to address at least a part of the 28% of P. falciparum proteins that are still missing a domain identification and the protein architectures that should be enriched with new domains (among the 2394 single domain proteins, 1214 of them contain a domain that covers less than the half of the protein length, and for these proteins we expect multiple domains to lie together).
Some unannotated proteins in P. falciparum likely contain completely novel domains, which are evolutionarily unrelated to domains present in the existing domain family databases. If that is so, CLADE would not find them, at least not until existing databases grow enough to cover more of that domain space. CCMs and computational power. By using HPC, CLADE demonstrates to push the limits in annotation reached by current methods. The first step of CLADE, generating CCMs, is the highly expensive one. It is performed once for all genomes to be annotated. All CCMs used to realize P. falciparum annotation have been constructed in 3. 7 months of computer time by using 250 CPUs (and they are made publicly available). The two subsequent steps (2 and 3), dedicated to genome annotation, are relatively fast.
For example, CLADE steps 2 and 3 ran in about 1 hr on 100 CPUs for the entire P. falciparum genome. CLADE domain library is expected to be regularly updated on new domains appearing in databases. This means that only CCMs for new domains need to be constructed and added to the existing library. This step can be realized independently from steps 2 and 3. In years to come, the expected improvements in HPC and in CLADE implementation (with a thoughtful selection of domain models) will render CLADE more computationally accessible.
Materials and Methods Databases Our method extends Pfam, an important collection of protein domains, that has been widely used for annotating proteins with unknown function. We use Pfam v27 (Pfam 27 , downloaded from http://pfam. sanger. ac. uk), containing 14 831 protein domains. In order to assess the performance of our method, we apply it to the set of all P. falciparum proteins. For this, we use PlasmoDB (http://PlasmoDB. org), that is the official repository of the P. falciparum proteins used as a reference database by malaria researchers. PlasmoDB v11. 1 contains 5 542 proteins. We used the UniProtKB database  [56] : 1. to extract NCBI taxonomy for sequences and the list of known domain architectures. 2. to recover the Pfam domain organization of all proteins in UniProt 15. 6 (Swiss-Prot 57.
6 and SP-TrEMBL 40. 6); we downloaded the dataset ftp://ftp. ebi. ac. uk/pub/databases/Pfam/releases/Pfam27. 0/swisspfam. gz and used it to analyze cooccurrence in CLADE annotations. (In our tables and figures, "Cooc", abbreviating "predictions supported by domain co-occurrence", and "Cooc-CCM", abbreviating "predictions identified by a CCM and supported by domain co-occurrence", count architectures whose domain pairs belong to already known architectures. ) A reference list of clades has been extracted from NCBI (http://www. ncbi. nlm. nih. gov/ taxonomy) and used for selecting a representative set of sequences in the construction of CCMs. Clades have been specified for Bacteria, Archaea, Viruses and Eukaryotes in S1 Table. We used the SCOP v1. 75  [41]  database to compare CLADE, based on the multi-source strategy, with the mono-source strategy of HMMScan  [2, 33]  and HHblits  [17].
Also, the SCOP datasets allowed us to compare CLADE with the computational strategy employed in SUPERFAMILY, a system that builds multiple hidden Markov models, for each protein superfamily, to realize sequence search. The SUPERFAMILY sequence search method is built on 1 962 superfamilies (from classes a to g), while CLADE relies on Pfam 27 containing 14 831 protein domains. To realize the comparative analyses, we considered SCOP domains whose associated sequences are coming from at least 10 species, and constructed a three testing sets from the ASTRAL95 dataset, containing 255 domain families and 8 633 sequences with at most 95% sequence identity, from ASTRAL30, made of 66 domain families and 1 251 sequences with at most 30% sequence identity, and from ASTRAL10 made of 18 domain families and 306 sequences with at most 10% sequence identity.
ASTRAL95, ASTRAL30 and ASTRAL10 are subsets of SCOP  [21]  and can be downloaded from http://scop. berkeley. edu/astral/subsets/ ver=1. 75. Tests with HHblits required the use of HHdb, a database built from the UniProt and NCBI NR databases, and provided in the HH-suite package  [17, 42].
Tools run for comparison HMMScan was run with default parameters and curated inclusion thresholds. The option -cut_ga, for model-specific thresholding (using profile's GA gathering cutoffs to set all thresholding), was used. HMMScan is included in the HMMer 3. 0 package  [33]  downloadable at http://hmmer. janelia. org/software. HHblits was run with default parameters. It is part of the HH-suite package-version 2. 0. 15  [17, 42]  downloadable at https://github. com/soedinglab/hh-suite.
Detailed description of the CLADE pipeline First, we detail how Pfam methodology works and how we modify it by including additional models, called clade-centered models (CCM). Then, we describe how to combine those models to produce reliable predictions. In the final step, we apply DAMA, an algorithm especially designed to take into account domain co-occurrence, to find the most likely domain architecture. Pfam methodology. Let D i be an arbitrary protein domain in the Pfam database. In Pfam, D i is associated to two sets of sequences, SEED and FULL. SEED is a set of protein sequences, called seed sequences for the domain D i , that share evolutionary and structural properties. It is defined by Pfam curators and results in a high-quality set of sequences. Pfam aligns seed sequences to build a profile hidden Markov model, SCM i , that represents the consensus of the seed alignment, that is the common features of the seed sequences.
SCM i is used by Pfam to scan databases of proteins with unknown function by localizing regions in sequences that are similar to the domain D i. Pfam predictions are those domain hits that HMMScan identifies with the -cut_ga option. In contrast, FULL is a automatically generated set. To construct it, Pfam uses SCM i to search in UniProtKB  [56]  for all significant hits from HMMScan (using the -cut_ga option). Most FULL sequences are never evaluated by experts, and some of them are probably false positives from HMMScan (sharing neither evolution nor structure). FULL contains SEED. In what follows, FULL is denoted S i ¼ fs i 1 ;. ; s i n g and SEED is denoted Clade-centered models. SCM i is a probabilistic model explaining how sequences in the SEED dataset S i Ã of Pfam 27 , associated to the domain D i , have evolved.
By contrast, we shall build a number of new models by exploiting information coming from (possibly different) evolutionary paths associated to specific species. They are called CCM i j , where CCM stands for clade-centered model, i indexes the domain D i , and j is an index running over the different models. To construct them, we take at most 350 sequences belonging to different species by choosing them from the FULL dataset S i in Pfam 27. The algorithm extracts a comparable number of sequences, whenever possible, from each clade. This automatic selection guarantees that the distribution of species is as uniform as possible across the panel of phylogenetic clades. (See list of clades leading the selection in S1 Table. ) This set of selected sequences is referred to as S i. Each sequence in S i will be used as a query to search for similar sequences, with PSI-BLAST  [52] , within the non-redundant protein database (NR).
From the resulting set of similar sequences, a probabilistic model CCM i j is constructed. As a result, we produce models CCM i 1 ;. ; CCM i m i , with m i ¼ jS i j j 350. See S1 Fig (  dotted lines ). Combining Model Predictions. Reliable predictions are obtained by combining domain hits from different models. We follow three main steps: 1. The building phase of the ensemble of clade-centered models. We modify Pfam original library, in such a way that, a single domain D i is now represented by an ensemble of models C i ¼ fCCM i 1 ;. ; CCM i m i ; SCM i g. We have used PSI-BLAST  [52]  and HMMer  [33]  for providing models in C i. The two tools exploit different data: individual sequences in the FULL dataset of Pfam 27 were used as queries to build CCM i j with PSI-BLAST, and sequences in the SEED dataset were used by HMMer in Pfam 27 to build SCM i.
First, the high divergence of the sequences in a protein dataset (for instance, the P. falciparum protein dataset) can imply that the distribution of confidence scores in the training set is different from the distribution in the testing data. Second, our models are very heterogeneous, since they were trained from different data sources, thus, it is not expected that there is necessarily an agreement among their predictions. Because of these two observations, any decision strategy that tries to find a perfect agreement of model answers is expected to fail. To avoid this, we designed meta-features which aim: (i) to highlight individual model results, when agreement among CCMs and SCMs models is not observed, and (ii) to provide an indication of the performance of all models. See schema in S2 Fig.
This was done with The Sequence Manipulation Suite software  [60]. To construct the negative training set, we randomly choose 20 sequences in each set S k , where k = 1. 14 831 (the number of Pfam domains), and shuffle their 2-mers. We obtain about 300 000 sequences. Notice that when we compare models of D i against these 300 000 sequences, only a few of them give an E-value smaller than 1 (chosen as threshold) and this ensures these sequences to be negative for D i. We repeat the construction 10 times and randomly choose, among all generated sequences, 1 000 sequences that will form the negative set. Positive and negative datasets have the same size, in order to avoid unbalanced sets. (For certain protein families this is obtained by using the supersampling SVM option. ) 3. The prediction phase.
SCMs were downloaded directly from the Pfam web site, while CCMs were built with PSI--BLAST (v. 2. 2. 28 for 5 iterations and E-value cut-off 1e-03) on the NR database (download in March 2014). PSI-BLAST was applied to sequences coming from species that were representatives of the whole tree of life. In its second step, CLADE searches for domains occurrences in the query sequences, based on CCMs and SCMs. In order to detect a large number of potential Pfam domains, we set a permissive search E-value (= 1) in PSI-BLAST and HMMer search, using default values for the remaining parameters. We combined the predictions obtained by all profiles by training a SVM from features coming from profile outputs. For SVM, we used the LIBSVM tool  [64]  (v. 3. 0) with default parameters, and we turned on the option "-b" to provide probability estimates.
Experiment on SCOP datasets To realize the comparison experiment on the three SCOP datasets, we used a leave-one-out strategy as follows. Given a domain family F D in one of the ASTRAL datasets, we considered the set of n sequences, coming from different species and associated to F D in ASTRAL, to create n test-sets for F D. Each test-set takes n -1 sequences for training and leaves one sequence out for the test. Then, we tested whether the sequence that was left out could be annotated by a model (or models) constructed without using it, and counted the correct identification of the domain as a true positive (TP), the identification of an erroneous domain as a false positive (FP) and the identification of no domain as a false negative (FN). Note that a domain is "correctly" identified when it belongs to the domain family F D.
This same procedure, was implemented for all systems we wanted to compare: HMMScan, HHblits, CLADE, CLADE BEv , CLADE_HHblits, CLADE BEv HHblits. For HMMScan (run with default parameters), each test-set of n -1 training sequences was first aligned with Clustal W (version 2. 1)  [65, 66]. Then we used hmmbuild to construct a probabilistic model (a profile) from the alignment. To compare profiles and test sequences we used hmmsearch that employs a composition bias filter by default for eliminating false positives. All tools can be found in the HMMer package-version 3. For HHblits (run with default parameters), we also constructed a profile from each test-set of n -1 aligned sequences (it is the same alignment as for the HMMScan experiment) but using hhmake. In HHblits experiment, test sequences must be represented by profiles and we used hhblits on HHdb (the HHblits database) to construct them.