dataesr/scientific-paragraphs-categorization huggingface dataset:
TOUGH-M1 Dataset TOUGH-M1  (Govindaraj and Brylinski, 2018 ) is a dataset of 505,116 positive and 556,810 negative protein pocket pairs defined from 7,524 protein structures. Pockets are defined computationally with Fpocket 2. 0 (Le  Guilloux et al. , 2009)  and filtered to include only predicted cavities having the greatest overlap with known binding residues, see Section 2. 1. As the TOUGH-M1 dataset is used for both training and evaluation but has not been used in a machine learning setting previously, we first define a sensible data splitting strategy before we report our results and compare to several baseline methods. Training and evaluation strategy The definition of independent subsets of data, as desired for meaningful evaluation of machine learning-based methods, is not straightforward in this case.
Indeed, a number of recent works have commented on the need for robust splitting methodologies when working with protein structure data  (Li and Yang, 2017; Feinberg et al. , 2018). A single protein structure may take part in multiple pairwise relationships, some possibly being in the training set and some in the test set, leading to a potential for information leakage. In addition, TOUGH-M1 may contain multiple structures representing a single protein family. Here, we propose to split at the structure level (instead of pairs), devoting 80% of structures for training and hyperparameter tuning and reserving the remaining 20% (1,457 structures, 40,296 pairs) as a hold-out test set. Any pairs connecting training and testing protein structures are discarded. Structures assigned to a common UniProt accession number (determined using the SIFTS database  (Velankar et al.
, 2012) ) are always allocated to the same subset. 220 PDB IDs to which no UniProt accession number could be assigned were removed. After hyperparameter search rounds, we retrain our network on the whole training set (5,847 structures, 628,234 pairs). Following  Govindaraj and Brylinski (2018) , the performance is measured with the Receiver Operating Characteristic (ROC) and the corresponding Area Under the Curve (AUC) is reported. To estimate the sensitivity to the choice of particular splits, we use repeated random sub-sampling (Monte Carlo) validation  (Dubitzky et al. , 2007). Having finalized the hyperparameters, we repeat splitting and training on ten random permutations of TOUGH-M1 and measure the standard error over the respective test sets. Baselines We compare DeeplyTough to three alignment-based methods chosen by the authors of the TOUGH-M1 dataset and to an additional alignment-free approach.
Results The measured ROC curves for the TOUGH-M1 dataset are shown in Figure  2. DeeplyTough achieves an AUC of 0. 933, outperforming by far all other approaches and achieving a substantial improvement over the second best performing method, SiteEngine (AUC 0. 733). The remaining two alignment-based methods, G-LoSA and APoc, achieve greater AUCs than PocketMatch, the alignment-free approach, with AUCs of 0. 683, 0. 662 and 0. 611, respectively. Furthermore, the performance of all methods is fairly stable across different test (and training) sets, with DeeplyTough achieving the lowest standard error. Analysis of the TOUGH-M1 positive pocket pairs that were assigned large distances (false negatives) highlights potentially ambiguous ground truths in the dataset. In particular, the bound ligands of false negative pockets show enrichment of biologically versatile endogenous molecules such as nucleotides (ATP, ACO), amino acid monomers (TYR, PRO, ASP) and sugars (GLC, NDG), as well as a number of non-biologically relevant ligands involved in the production of protein structure data such as crystallization agents (MPD, MRD), purification agents (CIT, FLC) and buffer solutions (TRS, B3P).
these pocket pairs do represent instances where related ligands are bound to unrelated proteins (constituting the definition of a positive pocket pair), we argue that in some cases there is limited structural similarity between pockets, and shared binding may be attributed to the conformational flexibility  (Haupt et al. , 2013)  or non-specificity of the bound ligand. These cases may be considered a limitation of the current dataset. On the other hand, analysis of TOUGH-M1 negative pocket pairs that were assigned small distances (false positives), revealed a large interconnected network of pocket pairs that were predicted similar. Inherently, there is a high potential for false positives in pocket pair datasets since the absence of common bound ligands in extant data does not render it impossible for two binding sites to bind related ligands.
Of the false positive pocket pairs examined, we found that many pockets were bound to polar ligand moieties containing anionic groups such as phosphates (2P0, T3P, S6P), sulfonamides (E49, 3JW), and carboxylates (G39, BES). Furthermore, false positive pockets seem to be enriched with polar residues suggesting that there may be similarity between these pockets, despite their negative labels. DeeplyTough performs well on the TOUGH-M1 dataset, however, it must be remembered that this is a trained method, and while results are calculated for data held-out from training, the data is still drawn from the same underlying distribution. It is therefore interesting to evaluate the performance of DeeplyTough on further datasets constructed independently.
Vertex Dataset The Vertex dataset introduced by Chen et al. (2016) comprises 6,598 positive and 379 negative protein pairs defined from 6,029 protein structures. The protocol for annotation of protein pairs derives from commonalities (or lack thereof) among experimentally measured ligand activities. In this benchmark, predicted protein-level similarities are obtained from a set of pocket-level similarities. In particular, the smallest of k × l predicted pocket distances is assigned to each protein pair of interest with k ligand binding sites collected across different PDB structures for one protein and l for the other. For the Vertex dataset, this amounts to 1,461,668 positive and 102,935 negative pocket comparisons in total. Unlike in TOUGH-M1, where binding sites are obtained from predicted cavities, the Vertex dataset defines pockets using their bound ligands directly.
Specifically, we define a pocket as all complete protein residues with any atom falling within 8 Å of any ligand atom. Training and evaluation strategy The network is trained on the whole TOUGH-M1 dataset. However, to prevent information leakage, we discard all structures having their UniProt accession number found in the Vertex dataset as well as 220 PDBs to which no UniProt accession number could be assigned, resulting in 6,979 structures and 820,140 pairs left for training. Following  Chen et al. (2016) , we measure performance with the ROC curve and corresponding AUC. Baselines We compare our approach to SiteHopper  (Batista et al. , 2014) , a structure-based pocket matching method chosen by the authors of the dataset. SiteHopper is an alignment-based method which represents binding sites as sets of points describing the molecular surface and nearby physicochemical features, which are aligned by maximizing the overlap of point-centered Gaussian functions.
Results The measured ROC curves are shown for the Vertex dataset in Figure  3. We can see that both SiteHopper (AUC 0. 887) and DeeplyTough (AUC 0. 818) achieve good performance on the dataset, while the gap to PocketMatch is large (AUC 0. 604). Importantly, the result indicates that our method generalizes well across two different methods for defining binding site geometric centers (computational and ligand-based). These results also hint that the ground truth pocket similarities in TOUGH-M1 and Vertex are comparable, despite being derived by different protocols.
ProSPECCTs Datasets ProSPECCTs  (Ehrt et al. , 2018)  is a collection of ten benchmarks recently assembled to better understand the performance of pocket matching for various practical applications. As for the Vertex dataset above, binding sites are defined by their bound ligands and we include complete protein residues with any atom that falls within 8 Å of any ligand atom. Training and evaluation strategy The network is trained on the whole TOUGH-M1 dataset. To prevent information leakage, we discard all structures having their UniProt accession number found in any of the ProSPECCTs datasets as well as 220 PDB IDs to which no UniProt accession number could be assigned, resulting in 6,369 structures and 715,520 pairs left for training. Following  Ehrt et al. (2018) , the performance is measured with ROC curves and the corresponding AUC.
Results The measured AUC scores for the ProSPECCTs datasets are given in Table  2  and compactly visualized in Figure  4. Dataset P1 evaluates the sensitivity to the binding site definition by comparing structures with identical sequences which bind to chemically different ligands at identical sites. Dataset P1. 2 measures this exclusively for chemically similar ligands. By reaching AUCs of 0. 96 and 0. 97, respectively, we can conclude that DeeplyTough is fairly robust to varying pocket definitions, which may be attributed to our stability loss as well as our data augmentation strategy. In Table  3 , we observe that the stability loss alone is responsible for more than AUC 0. 1 difference across multiple ProSPECCTs datasets. In addition, the box plot in Figure  5  illustrates a clear distance-dependent distinction between identical and non-matching binding site pairs, likely a virtue of our margin-based contrastive loss function.
Dataset P2 assesses the sensitivity to binding site flexibility by comparing the pockets of Nuclear Magnetic Resonance (NMR) structures with more than one model in the structure ensemble. DeeplyTough achieves AUC 0. 93, which indicates a slight susceptibility to the conformational variability of proteins. We believe this could be addressed by introducing an appropriate data augmentation strategy in the training process. Next, two decoy datasets evaluate the discrimination between nearly identical binding sites differing by five artificial mutations leading to different physicochemical properties (Dataset P3) or both physicochemical shape properties (Dataset P4). Performing at AUC 0. 72, DeeplyTough has some difficulty ranking original binding sites pairs with identical sequences, higher than pairs consisting of an original structure and a decoy structure.
This suggests that the learned network might be overly robust and may not pay enough attention to modifications in the sites. When compared to existing approaches, however, our approach ranks proportionally well as fifth and eighth, respectively. Moreover, the performance is well correlated with the number of mutations in the sites  (AUC 0. 55, 0. 63, 0. 65, 0. 72 and AUC 0. 53, 0. 60, 0. 66, 0. 68 P4, respectively), consistent with the intuition that pockets that have been more heavily mutated should be easier to differentiate. A further two datasets contain sets of dissimilar proteins binding to identical ligands and cofactors. Datasets P5 and P5. 2 have been compiled by  Kahraman et al. (2007)  and contain 100 structures bound to one of ten ligands (excluding and including phosphate binding sites, respectively).
Datasets P6 and P6. 2 contain pairs of unrelated proteins bound to identical ligands, assembled by  Barelier et al. (2015)  (excluding and including cofactors, respectively). Our method scores better on the kahraman dataset (AUC 0. 66) than on unrelated proteins (AUC 0. 59), consistent with reports that the kahraman dataset represents an easy benchmark since the chosen ligands may be distinguished solely by their sizes  (Hoffmann et al. , 2010). Finally, Dataset P7 is a retrieval experiment measuring the recovery of known binding site similarities within a set of diverse proteins. DeeplyTough reaches AUC 0. 80 (average precision 0. 44), which places among the best performing half of the baseline approaches. In summary, our method performs consistently well across ProSPECCTs datasets although it does not attain the best performance for any individual task.
For practical applications, we suggest these results support the use of DeeplyTough as a fast basic universal tool, rather than a specialist one. Running time In addition,  Ehrt et al. (2018)  published the running times for each algorithm on Dataset P5 (100 pockets and 10,000 comparisons). The runtime for DeeplyTough is 206. 4 seconds in total, where the preprocessing with AutoDock 4  (Morris et al. , 2009)  and HTMD  (Doerr et al. , 2016)  requires 191. 4 seconds (serialized on a single CPU core) and the descriptor computation and comparison takes 15 seconds on an Nvidia Titan X. This makes ours the fourth fastest approach in the benchmark, behind PocketMatch, RAPMAD, and TM-align. SiteHopper is a slower approach with a total runtime of 3982. 6 seconds (17th of the ProSPECCTs baseline methods).
For DeeplyTough, we expect that even further reduced runtimes may be achieved through full parallelism of the initial preprocessing. Protein binding site space T-distributed Stochastic Neighbor Embedding (t-SNE)  (Maaten and Hinton, 2008)  can be used to visualize the learned descriptor space obtained by DeeplyTough. Figure  6  shows the embeddings of pockets in Dataset P1 colored by their UniProt accession numbers. For the most part, protein pockets derived from the same protein family are clustered together, suggesting that the network embeds similar pockets close to each other in the descriptor space. Similar conclusions can be drawn for protein pockets derived from the Vertex dataset in Figure  7 , wherein pockets are colored by their respective toplevel SCOPe classifications  (Fox et al.
Training Data Ablation Deep learning models are proverbially known to require large amounts of data for training. To provide an insight into the dependence of DeeplyTough on the large scale of the TOUGH-M1 dataset, we experiment by artificially limiting the training dataset size using two approaches, in both cases validating on ProSPECCTs as an independent set. For simplicity, all training and network hyperparameters are kept fixed. First, we investigate restricting the number of relationships the network is allowed to see. Thus, pairs of random subsets of size between 1,000 and 100,000 are sampled from the positive and negative set of pocket pairs present in the original training set. The results shown in the upper part of Table  3  suggest the network does not strongly suffer from the removal of training data in this way, even if the training set is smaller by two orders of magnitude.
We expect the likely cause of this is that even for a reduced training set of only 2,000 pairs, the effective number of structures is still relatively high (about 2,200 PDBs). Hence, we look into constraining the pocket variability in the data. Random subsets of size varying between 1,000 and 3,000 are sampled from the original 6,369 structures available for training and only pairs lying within these subsets are retained. The results shown in the bottom   3  indicate that the performance starts to severely deteriorate once the number of structures drops below 2,000, even if this corresponds to about 70,000 induced pairs. Therefore, we may conclude that for our method, pocket diversity in the data is relatively more important than the number of ground truth relationships. This observation suggests that it may be appropriate to construct new pocket matching datasets using as many structures from the PDB as possible, even if relatively few pocket pairs are defined.
Finally, we reverse the settings and train on ProSPECCTs while validating on the TOUGH-M1 and Vertex datasets (training on 77,665 pairs over 1,395 structures for the former and on 30,175 pairs over 1,246 structures for the latter). This results in major degradation of AUC scores, 0. 740 on TOUGH-M1 (versus 0. 933 when training on TOUGH-M1) and 0. 709 on the Vertex dataset (versus 0. 817). To summarize, it appears that the large scale of TOUGH-M1 is a necessary condition for our method to perform well. However, we expect pre-trained DeeplyTough to be amenable to successful fine-tuning for smaller task-specific datasets.
Conclusion In this work we have proposed a deep learning-based approach for pocket matching. DeeplyTough encodes the 3D structure of protein binding sites using a CNN into descriptors such that the similarity between binding sites is reflected in the Euclidean distances between their descriptors. Once a set of descriptors is computed, pocket matching is simple and efficient, without any alignment operation taking place. In a thorough evaluation on three benchmark datasets, we have demonstrated excellent performance on held-out data coming from the training distribution (TOUGH-M1) and competitive performance when the trained network needs to generalize to independently constructed datasets (Vertex, ProSPECCTs). We have taken advantage of several recent innovations such as rotationally and translationally invariant CNNs, data augmentation and the inclusion of an explicit stability loss function to encourage robustness of the network with respect to nuisance variability of the input.
for one to four mutations in Dataset P3 and 2. AUC values for 22 pocket matching methods on each of ten ProSPECCTs datasets. P1 P1. 2 P2 P3 P4 P5 P5. 2 P6 P6. 2 P7 Cavbase 0. 98 0. 91 0. 87 0. 65 0. 64 0. 60 0. 57 0. 55 0. 55 0. 82 FuzCav 0. 94 0. 99 0. 99 0. 69 0. 58 0. 55 0. 54 0. 67 0. 73 0. 77 FuzCav (PDB) 0. 94 0. 99 0. 98 0. 69 0. 58 0. 56 0. 54 0. 65 0. 72 0. 77 Grim 0. 69 0. 97 0. 92 0. 55 0. 56 0. 69 0. 61 0. 45 0. 65 0. 70 Grim (PDB) 0. 62 0. 83 0. 85 0. 57 0. 56 0. 61 0. 58 0. 45 0. 50 0. 64 IsoMIF 0. 77 0. 97 0. 70 0. 59 0. 59 0. 75 0. 81 0. 62 0. 62 0. 87 KRIPO 0. 91 1. 00 0. 96 0. 60 0. 61 0. 76 0. 77 0. 73 0. 74 0. 85 PocketMatch 0. 82 0. 98 0. 96 0. 59 0. 57 0. 66 0. 60 0. 51 0. 51 0. 82 ProBiS 1. 00 1. 00 1. 00 0. 47 0. 46 0. 54 0. 55 0. 50 0. 50 0. 85 RAPMAD 0. 85 0. 83 0.
A step further is taken by the study of  Alberdi et al. (2019) , which uncovers the insufficiently harmonised Habitats Directive reporting procedures, resulting in the existence of different national forest habitat type definitions, different ways of their identification in the field, and in diverse interpretations of structural and functional indicators and their utilisation in the conservation status assessments. The same study also provides an overview of the use of the most common NFI indicators representing structural and functional aspects of biodiversity. Large and small-scale forest and landscape inventories, as well as specific monitoring schemes, are common ways for collecting data about forests, needed in biodiversity and the assessments of forest habitat type conservation status.
g. , grasslands with sparse autochthonous trees; natural afforestation of agricultural lands; Forest Europe, 2019) or similar land uses. Statistically, NFI datasets represent samples that are regarded as realisations of stochastic processes. The data of these samples thus allow deriving parameters and inferences for populations and their strata. Perhaps the most critical factors are the inventory design and its implementation over time because they define the quantity of the data, the precision of estimates, and the timeliness of the data. While the continuous panel-based NFI provides the data from sub-samples yearly, from which timely estimates can be derived, the periodic NFI provides less frequent but more precise estimates, which are derived from a sample that is usually surveyed within a rather short period (e.
g. , within a single or few years). However, whenever estimates for specific years are needed, they are obtainable either by interpolation or modelling. In addition to the limitations of sampling designs, NFI biodiversity data also have methodological shortcomings due to incompletely standardised data collections and observed variables. Comparisons between countries show that the data often differ in terms of variable and indicator definitions, time-series completeness, measurement protocols, and thresholds  (Chirici, Winter, & McRoberts, 2011; Winter et al. , 2011). To improve their usability, some variables, such as deadwood  (Rondeux et al. , 2012; Woodall, Rondeux, Verkerk, & Ståhl, 2009) , naturalness  (McRoberts, Winter, Chirici, & LaPoint, 2012) , and ground vegetation  (Alberdi, Condés, & Martínez-Millán, 2010; Alberdi, Condés, McRoberts, & Winter, 2018)  are subject to harmonisation studies.
Conversely, the statistical analysis aimed at investigating the initial indicator data distributions (Table  1 ). The joint database (in text: Joint database) was developed with the NFI data of Italy, Slovenia, and Spain. For some reasons (e. g. , nonexistence, non-comparability, scarcity of event) some data were unavailable. Thus, we only explored the data distributions of those indicators whose data occurrences were common (Table  4 ). The joint database contained the data of the 9110 Luzulo-Fagetum beech forests and 9530 forest habitat type Sub-Mediterranean pine forests with endemic black pines, regardless of whether the NFI data belonged to the Natura 2000 network or not (Table  5 ). The number of NFI records (n) between the three countries differed. The differences were due to different national sampling grids and national forest habitat type areas.
Tree species-based indicators uphold the conservation status components of range and area, structure, and prospect (Table  6 ). While the components of structure and prospect are supported satisfactorily, the indicators supporting the area and range are underdeveloped. One of them is forest habitat type identity, which helps assess the total forest and forest habitat type area, the size of individual forest habitat type complexes and the degree of their mingling with neighbouring forest and non-forest habitat types. The indicator is powerful at landscape (above individual habitat types) and large landscape scales. Tree-species-based indicators are generally easily collected by the existing NFI data collection protocols. Conversely, collecting ground vegetation data is complex, and the protocols have yet to be improved  (Alberdi, Condés, & Martínez-Millán, 2010).
Indicator quality Apart from the forest habitat type, which is generally determined via phytosociological analysis  (Alberdi et al. , 2019) , tree species-based, development stage and tree-dimension-based indicators are obtainable from all NFI datasets. These indicators met most suitability requirements (Table  7 ) because of their consistent definitions and good scores of other quality dimensions. Sapling density (small trees; measured at breast height), however, should be used carefully because ground vegetation data and the data about saplings below breast height are usually missing. The situation with the development stage and stand age is similar. The considerable variability of tree diameters in the stands, managed by the improved shelterwood systems, also occasionally hampers firmly determining the development stage and stand age.
Some differences are also due to the sampling designs. In the view of the Habitats Directive Article 17 reporting, NFIs should deliver rather fresh data to be able to detect changes in conservation status between two occasions and over time. For this reason, periodic NFIs (i. e. , inventories which perform measurements on the whole sample within an NFI cycle) with relatively short NFI cycles and continuous panel-based NFIs, delivering data annually, would be the most suitable. Notwithstanding, some changes also should be made in the Habitats Directive reporting system. One of the possibilities would be to harmonise its reporting cycle with the reporting cycles of other processes, for which many NFI data and indicators are prepared. In view of forest habitat type assessments, the statistical approaches may produce two sorts of outcomes.