dataesr/scientific-paragraphs-categorization huggingface dataset:
MODIS validation For each image in the full dataset, we extracted the corresponding fractional snow cover value from MODIS products MOD10A1 and MYD10A1  [41]. These products are derived from radiance data from the Terra and Aqua satellites respectively and use a 500-meter resolution. Using values from both satellites increases the temporal coverage of MODIS data. We considered snow to have been detected if either the MOD10A1 or the MYD10A1 product had a fractional snow cover greater than zero. If neither product reported fractional snow cover corresponding to a given PhenoCam image (e. g. due to cloud cover), then we considered that image to have no MODIS information about snow and did not use it for the following analyses. A total of 96,151 images from the set of all images without a crowd consensus of 'bad image' (56%) had a MODIS classification of snow or no snow.
When images with a crowd consensus of 'bad image' were removed from the gold standard dataset, the crowd consensus of 'snow' vs. 'no snow' had an accuracy of 99. 1% compared to the gold standard (1797 images) for all sites and 99. 4% for the datasets consisting of Type I and Type II sites (1352 images) and Type I sites only (924 images), respectively. (See S1 Appendix for confusion matrices. ) These results indicate that the crowd consensus classifications are of excellent quality and can be used as reliably as an expert-annotated dataset.
Deep learning classification Accuracy for the convolutional neural networks (CNNs) compared to the crowd consensus varied from 83. 2% to 97. 5%, depending on the specification of training and validation sets (Table  1  and S1 Appendix). Using the top-5 categories from Places365-VGG directly, accuracy was 83. 2% for the full dataset. Transfer learning increased the accuracy to 97. 0% when images were fully randomized between training and testing sets. The apparent upper bound was only slightly higher at 97. 1%, suggesting that the CNNs with transfer learning would be excellent at predicting whether snow is present or not for new images from existing sites. Transfer leaning increased accuracy from 83. 2% to 91. 6% when images assigned to training and testing sets were split by site. This accuracy is lower than when training and testing sets were fully randomized and reflects the challenge for the CNNs in predicting whether snow is present or not at sites it has not yet seen.
These new sites are outside the set of sites on which the CNN trained, and so the CNN is making inferences about images at these new sites based only on what it knows snow looks like at other sites. When a standard camera configuration was used for all the sites in the dataset (i. e. the subset datasets), the CNNs performed better on sites they had not yet seen; the difference in accuracy was 5. 4% between the cross-validation models for the whole dataset, but just 3. 5% and 3. 7% for the two subsets. CNNs trained on different sets of images almost always agreed on the snow classification for individual images. For the CNNs trained to predict new images from existing sites, the agreement rate was 99. 4% and for those trained to predict images from new sites, the rate was 97. 0%. For the small number of images where there was disagreement, the reason was usually because they were sites with small amounts of patchy snow.
These images frequently also had non-unanimous consensus by participants. Accuracy was always lowest for the full image dataset (Types I, II, and III), but by less than a few percentages points compared to the subset datasets. This suggests that a standard configuration across cameras has a small, but measurable impact on the ability of CNNs to learn to identify snow. Accuracies for the two subset datasets (Type I and II, Type I only) were similar, suggesting that choice of camera model does not affect the ability of CNNs to learn to identify snow if the cameras follow a standard configuration. Sensitivity (true positive rate) for the CNNs with transfer learning was lower than specificity (true negative rate; Table  2 ). Specificity ranged from 94% to near perfect, whereas sensitivity for the most accurate CNN was 90.
5% (S1 Appendix). This means that the CNNs fail to detect snow that is present ("missed snow") much more often than they falsely detect snow that is not there ("phantom snow"). The overall CNN accuracies, therefore, reflect the high specificity, because images without snow are more frequent. We examined the reasons for misclassification by the CNNs. Snow was missed by the CNNs most often when there was little snow in an image and the snow was patchy (Fig  2 ). These "missed snow" images also appear to be difficult for humans to consistently classify. Of the "missed snow" images, 43% of them have non-unanimous classification, as compared with 6% for the whole dataset. Some of the errors were also due to human misclassification of images that do not have snow. There were few errors when snow was not present, and classification of images without snow was almost perfect in predicting new images from existing sites.
Discussion Using crowdsourcing and deep learning in conjunction with a network of automated near surface cameras provides an automated and accurate means by which the distribution of snow can be estimated, at high temporal frequency (daily), fine spatial resolution (10-100 m), and across a broad spatial extent (regional-to-continental). Crowdsourcing was as good as expert labeling for the task of identifying snow presence or absence in digital images and resulted in a high-quality human-labeled dataset. Using transfer learning and this dataset, we were able to train deep neural networks to determine the presence or absence of snow at 133 heterogeneous sites with up to 98% accuracy. This novel processing chain-involving automated sensors, crowdsourcing, and transfer learning of deep convolutional neural networks (CNNs)-has the capacity to accelerate the acquisition and processing of a wide range of data types for environmental monitoring and research.
Volunteer classifications were as accurate as expert classifications when combined into a consensus classification, demonstrating that simple tasks such as determining 'snow' versus 'no snow' can require as few three independent volunteer classifications for high accuracy. More difficult or complex tasks might require more classifications or other means to ensure data quality  [47]. We crowdsourced labels to our entire image data set of 184,000+ images because we wanted a consistent and complete set of labels for a PhenoCam data product  [33]. However, we could have been more efficient. Using temperature records from the various sites would have allowed us to quickly classify many images as having no snow because it is too warm, leaving far fewer for participants to classify. If we had only been creating a dataset for training CNNs, we could have been even more efficient and subsampled-by eliminating warm-weather images and using imagery from every-other day, for example, as consecutive days frequently look very similar  [48].
Deep learning classification While the overall accuracy of the CNNs was very high, the accuracy for images without snow was about ten percentage points higher than the accuracy for images with snow. The main reason that the CNNs missed snow was that for some images, snow was present in small amounts or was patchy (Fig  2 ). Presumably, this patchy snow did not match the gestalt of what a snowcovered landscape should look like, based on the majority of the training data. In many cases, though, images with patchy snow were correctly classified. These images with patchy snow were also the ones for which there was most likely to be disagreement among CNNs trained on different datasets, suggesting that they lie close to the hypersurface that separates classes within the state space of CNN parameters  [49].
These white objects could be permanent or temporary, and included white vehicles, buildings, and pieces of equipment (Fig  3 ). CNNs that were trained to predict snow in new images from existing sites were familiar with these sites and did not mistake the white objects for snow. However, the CNNs that were not trained on the sites with white objects were not always able to determine whether or not the white objects constituted snow. In computer vision, it is assumed that the test set is drawn from the same distribution of images as the training set  [24]. When the test set constitutes a different distribution, errors are more likely to occur. However, because we used transfer learning, the CNNs should retain an underlying representation of white objects that are not snow from the original Places2 database  [52].
Conclusion We have demonstrated an automated method for creating location-specific, high temporal frequency, data on the presence or absence of snow using imagery from automated, near-surface camera monitoring sites. Though we use only about a hundred cameras that were originally set up for a different purpose, such a method shows potential for improving snow monitoring. While the individual cameras cover only a small spatial extent, they were selected to sample across a wide range of North American ecosystems and human land use. As a result, we expect the model to perform well generally for images of temperate North American outdoor scenes, and especially for those that were created using the PhenoCam camera protocol. The cameras are also necessarily limited in number, but the strength of their high accuracy and hourly images can be leveraged by combining camera data with data from technologies that cover high spatial extents, such as satellite and aerial imagery, and possibly crowdsourced image databases such as Flickr.
The uses for our dataset of labeled images are numerous. They could be used to validate MODIS snow products and to help refine their algorithmic approaches. Additionally, the dataset could be used as input to snow models that use data streams from satellites or airborne platforms or crowdsourced image databases to create more accurate continuous predictions of snow. For example, MODIS snow data could be used to create a model of snow extent and then data from surface-based cameras could be used to refine model parameters for higher accuracy or to fill in gaps inferentially when clouds prevent direct measurement. Because our data processing system is automated, such models could be run in near real time. Computer vision researchers might also use the variety of high-quality outdoor images in our labeled dataset as a baseline to better capture information from crowdsourced outdoor image datasets that are noisier in nature.
We expect that our general method of automated sensors, crowdsourced labels, and deep learning for data processing will be implementable for a broad range of complex environmental datasets. We have shown that this method can achieve up to 98% accuracy for automated detection of snow from stationary time-lapse cameras. Other types of automated imageryfrom animal camera traps, from remote sensors on satellites and airborne platforms, and from deep sea mapping cameras, for example-can also benefit from this approach. Deep learning has also been shown to be effective in processing audio  [61]  and video  [62]  streams, and so audio environmental data, such as animal acoustics, as well as video environmental data, such as animal behavior video monitoring, can potentially benefit from this method as well.
Acknowledgments We thank our Knowxel collaborators,  Angel Sappa  and  Ariel Amato  at the  Centre de Visioper Computador ,  Universitat Autònoma de Barcelona , and the many participants who contributed snow classifications. We appreciate the work of  Don Aubrecht ,  Steve Klosterman ,  Morgan Furze ,  Miriam Johnston , who provided expert Knowxel classifications, and to  Tom Milliman  for PhenoCam data management. This manuscript was improved through.
Computational protein docking is a powerful strategy to predict structures of protein-protein interactions and provides crucial insights for the functional characterization of macromolecular cross-talks. We previously developed InterEvDock, a server for ab initio protein docking based on rigid-body sampling followed by consensus scoring using physics-based and statistical potentials, including the InterEvScore function specifically developed to incorporate coevolutionary information in docking. InterEvDock2 is a major evolution of InterEvDock which allows users to submit input sequences -not only structures -and multimeric inputs and to specify constraints for the pairwise docking process based on previous knowledge about the interaction. For this purpose, we added modules in InterEvDock2 for automatic template search and comparative modeling of the input proteins.
The InterEvDock2 pipeline was benchmarked on 812 complexes for which unbound homology models of the two partners and co-evolutionary information are available in the PPI4DOCK database. InterEvDock2 identified a correct model among the top 10 consensus in 29% of these cases (compared to 15-24% for individual scoring functions) and at least one correct interface residue among 10 predicted in 91% of these cases. InterEvDock2 is thus a unique protein docking server, designed to be useful for the experimental biology community.
On the other hand, structural models can be obtained for a large fraction of proteins in interaction networks thanks to homology modeling  (36) , making them amenable to protein-protein docking. To date, most free docking servers, except the HDOCK server, allow users to provide only input structures but no input sequences for the protein partners. Based on the user-oriented considerations mentioned above, here we introduce the InterEvDock2 server which represents a major evolution over the original InterEvDock. Protein sequences can now be provided as input, and not only 3D structures. To handle sequence inputs, we have added a module that performs comparative modeling prior to docking based on an automatic template search protocol. In case the user has biological input such as a position that is known to be involved in the interface between the two protein partners or a pair of residues known to be in contact, restraints with a tunable distance threshold can be specified for use in the docking procedure.
This is crucial to ensure that all available biologically relevant information is used for InterEvDock2 predictions. In addition, InterEvDock2 implements the possibility to submit structures of oligomers as input to the pairwise free docking. Such an option is generally complicated in co-evolution analyses since the joint MSAs have to be generated for every chain of an oligomer. This process is now fully automatized in InterEvDock2, allowing users to submit inputs such as homodimers or more complex structures as that of the nucleosome made of ten subunits. InterEvDock2 also benefits from improved accuracy by integrating the most recent FRODOCK 2. 1 algorithm for rigid-body docking and scoring  (13)  and implementing an improved consensus selection and from a speedup in the generation of joint MSAs for the two protein partners.
The InterEvDock2 pipeline was benchmarked on 812 complexes from the PPI4DOCK database (37) designed to ensure unbiased evaluation of the performance of free docking from unbound homology models. A total of 29% of those 812 cases have an acceptable or better solution among the top 10 consensus models returned by InterEvDock2. As InterEvDock, InterEvDock2 also outputs a list of the 10 residues most likely involved in the interface and at least one residue was correctly predicted in 91% of the 812 benchmark cases.
Web interface Users are expected to provide for each protein partner either an input sequence or an input structure (Figure  1 ). Input structures can be uploaded or retrieved automatically from the Protein Data Bank (PDB) by typing in the PDB code and optionally one or more chain identifier(s). More options are available through the 'advanced options' menu (see Figure  2  and Supplementary Figure  S1 ). Optional breakpoints can be selected, either after template search to choose among up to 20 suggested templates prior to modeling, or after modeling for interactive visualization of the models prior to docking. When input sequences are provided, users can specify which template to use for homology modeling; as for structure inputs, the template can be uploaded or directly retrieved from the PDB.
Users can optionally specify the distance that will be   (17) , except that the FRODOCK algorithm was updated to version 2. 1 (13) and the consensus calculation was slightly modified to save time and improve results. New features are available allowing the user to provide only an input sequence for one or both partners: (i) if the user does not provide a template, search for a suitable template using HHsearch  (38, 39) ; (ii) if the user provides a template but no query-template alignment, alignment of query sequence with template sequence using MAFFT (40); (iii) once a template and a query-template alignment are available for each partner with no user-provided structure, comparative modeling using a RosettaScripts (41) protocol based on RosettaCM  (42)  to build a 3D model for (at least part of) the input sequence.
Once a 3D structure or a structural model is available for each partner: (iv) exhaustive sampling using the rigid-body method FRODOCK 2. 1; (v) (new feature) if the user provides information on residues (or pairs of residues) involved in the interface, applying constraints to filter sampled solutions; (vi) if the user does not provide a joint MSA for the two protein partners, co-MSA generation; (vii) clustering and scoring by three scores, FRODOCK 2. 1, SOAP-PP  (34)  and InterEvScore (  32  used for each constraint. An InterEvDock2 session identifier can also be provided in order to re-use docking results from a previous run and test different constraints. As in In-terEvDock, users may input the joint MSAs used for coevolution-based scoring; otherwise the joint MSAs will be built by the server through an automated procedure.
In case an oligomeric structure is submitted as one of the two docking partners, the joint MSAs will also be automatically calculated and processed by the server for every chain of the oligomer. A demonstration case using sequences as input to the docking (and optionally a constraint) is available from the InterEvDock2 submission page. The web page resulting from an InterEvDock2 submission contains information about the best ranked decoys, which can be explored interactively thanks to the PV We-bGL applet (M. Biasini, https://dx. doi. org/10. 5281/zenodo. 12620). Detailed results are available in a downloadable archive, also containing a script for easy loading and offline visualization of the best docking solutions with PyMOL (The PyMOL Molecular Graphics System, Schr ödinger, LLC, New York, NY).
The InterEvDock2 server benefits from parallelized implementation in the dedicated infrastructure built at RPBS and from data privacy ensured in the Mobyle framework. (17) , except that the FRODOCK algorithm was updated to version 2. 1  (13). In the final step (viii) a consensus list of 10 most likely models is calculated. Since decoys well ranked by at least two different scoring methods (out of the three methods used in InterEvDock2) have higher chances of being correct, the 3*top 10 models for each score are re-ranked according to the number of similar decoys (defined as ligand RMSD ≤ 10 Å) within the top 50 models of the other two scores (down to a minimum of two similar decoys). In case of a tie, priority is given to InterEvScore top 10 models, then SOAP-PP, then FRODOCK. If necessary, the consensus list is then filled up to 10 models by selecting the best models from each score (four from InterEvScore, three from SOAP-PP and three from FRODOCK).
(ii) If the user provides a template but no querytemplate alignment, the query sequence is aligned with the template sequence using MAFFT  (40). (iii) Once a template and a query-template alignment are available for each protein with no user-provided structure, comparative modeling using a RosettaScripts (41) protocol based on Roset-taCM (  42 ) is performed to build a 3D model for (at least part of) the input sequence. Due to runtime considerations, compared to the procedure used to build the PPI4DOCK database  (37) , the comparative modeling protocol implemented in the InterEvDock2 web server involves fewer optimization cycles (see protocol details in the Supplementary Methods). This protocol is quite robust for templates with relatively high homology but it can lead to loss of precision for more remote templates (typically when both templates have <50% sequence identity with the query proteins).
Benchmarking on PPI4DOCK To assess the predictive power of the InterEvDock2 server on 3D models, we have set up the most extensive benchmark to date, using unbound models as input of the docking simulations. The PPI4DOCK database (37) was designed to ensure unbiased evaluation of free docking performance and contains 1417 non-redundant heterodimeric docking targets based on unbound homology models. The InterEvDock2 pipeline was tested on the subset of 812 protein complexes from PPI4DOCK for which pairs of joint MSAs with more than 10 sequences could be obtained (excluding any antibody complex) and FRODOCK 2. 1 (13) was able to generate at least one acceptable or better decoy among the top 50 000 decoys. The list of the 812 complexes used for benchmarking and detailed results are provided in http://bioserv.
rpbs. univ-paris-diderot. fr/ services/InterEvDock2/table. html. This benchmark dataset is roughly an order of magnitude larger than other typical docking benchmarks, among which the widely used Weng benchmark  (43). For each of the 812 targets, PPI4DOCK provides unbound homology models of the two protein partners as well as the joint MSAs used for docking and scoring in the InterEvDock2 pipeline. As on the web server, the predictions for each case consist in the top 10 consensus interface models and the top 10 interface residues, which are used to assess the InterEvDock2 performance. A solution is defined as acceptable or better according to the criteria defined by the CAPRI consortium  (44). The prediction performance of InterEvDock2 is reported in Table  1. Among the 812 targets, 29% (239) have at least one model of acceptable or better quality in the top 10 consensus obtained from the InterEvDock2 pipeline, which represents a significant improvement over the top 10 success rates of the three individual scores used to build the consensus (see also Supplementary Figure  S3 ).
Analysis of InterEvDock2 performance depending on the minimum sequence identity between the target and template shows a moderate drop in success rate for models built with remote templates (<30% sequence identity) and an increased success rate for models built with very close templates (≥95% sequence identity), compared to the overall InterEvDock2 success rate (see Supplementary Table  S1 ). Direct comparisons with previous benchmarks are difficult because the benchmark dataset used here is much larger than other datasets typically used to assess docking and scoring performance. Comparison with previously reported success rates on the Weng benchmark  (17, 43)  are provided in Supplementary Tables  S2  and S3. An interesting feature of the Weng benchmark compared to PPI4DOCK is that it contains targets where one partner is multimeric.