dataesr/scientific-paragraphs-categorization huggingface dataset:
Dataset 1. Validation of dataset 1 is performed on the picture naming task (described in section 2. 1. 1) at both scalp and cortical level. First, data are preprocessed using Automagic Matlab toolbox  33. The preprocessed data are then uploaded to EEGLAB toolbox  34  , and the event response potential (ERP) is calculated over all 23 participants and trials. The ERP was segmented using a toolbox called "Microstate eeglab toolbox"  35. The segmentation was done using the K-means modified algorithm and results showed five topography states (Fig. 6a ). The first state (T1) starts from 1 to 94 ms, the second state (T2) from 94 to 154 ms, the third state (T3) from 154 to 227 ms, the fourth state (T4) from 325 to 374 ms and the fifth state (T5) from 443 to 500 ms. Globally the results show a transition from the occipito-temporal lobe (T2, visual processing) to frontal lobe (T3) with a repetition of this spatial pattern between T4 and the second appearance of T3.
Noteworthy, both these sequences are preceded by the T1 pattern that seems to correspond to a fronto-medial activation. Second, the EEGs data and MRI template were coregistered using Brainstorm toolbox  36  and Destrieux atlas used to parcellate the cortical surface onto 148 regions  37. The source time series are reconstructed using the weighted minimum norm estimated (wMNE) method available in Brainstorm. The averaged regional time series over all participants is then segmented using the k-means modified algorithm integrated in eeglab microstate toolbox. The automatic segmentation using the same tool as for scalp showed seven states (Fig. 6b ): the first state (S1) Fig. 4  Exemplary EEG-BIDS for one participant in dataset 1 (a) and dataset 2 (b). from 1 to 94 ms, the second state (S2) from 94 to 143 ms, the third state (S3) from 143 to 204 ms, the fourth state (S4) from 204 to 250 ms, the fifth state (S5) from 250 to 297 ms, the sixth state (S6) from 297 to 380 ms and the seventh state (S7) from 380 to 500 ms.
All three methods can be used to detect limits of cognitive processes, but cognition can't be reduced to a global mixture of signal nor to a specific frequency range. That is why the network method and the topographical method converge quite well on MS2 and T3 while the rises and falls of the amplitude of the sources makes it difficult to operate this division except when signal amplitude is at stake (transition S1 = >S2; T1 = >T2 at 94 ms). Dataset 2. Here we conducted the same analysis, as for the visual stimuli, on the auditory naming task. Participants were asked to name the objects they listen to. Scalp and source -level analyses are summarized in Fig. 8  and the source-space connectivity-based analysis (in the beta band) is illustrated in Fig. 9. Figure  8  shows that at both scalp and source level, an activation in the parietal and temporal cortex can be observed (T2 and S2).
Usage Notes Some of the most common software packages for analyzing these data are freely available, and include FieldTrip (http://fieldtrip. fcdonders. nl/), MNE (http://martinos. org/mne/), Brainstorm (http://neuroimage. usc. edu/brainstorm), EEGLAB (http://sccn. ucsd. edu/eeglab/), and EEGNET(https://sites. google. com/site/eegnetworks/), Automagic (https://github. com/methlabUZH/automagic). The code used to identify the modularity states is available at https://github. com/librteam/Modularity_algorithm_NN. This dataset has multiple potential uses for cognitive neuroscience and for methodological development in EEG analysis, such as: 1) The analysis of the brain dynamics at sub-second time scale in different conditions such as comparing picture naming vs. spelling. 2) The validation of several parameters involved in the methods used to estimate cortical brain networks (called EEG source connectivity) through scalp EEG, such as the open question of optimal number of channels.
Name agreement (%) 94,3243 95,8378 7,5058 5,8572 0,3370 Image agreement (average) 3,7550 3,6126 0,4481 0,7356 0,3189 Age of acquisition (average) 2,3292 2,5289 0,6152 0,5819 0,1557 Number of letters 5,9730 6,1622 1,9506 1,9792 0,6800 Number of phonemes 4,2162 4,2703 1,7502 1,6608 0,8920 Number of syllables 1,7297 1,7027 0,8383 0,7769 0,8860 Number of morphemes 1,0541 1,1351 0,2292 0,4191 0,3064 Oral frequency 12,1324 9,1903 13,9555 6,9085 0,2556 Written frequency 11,1673 11,0351 11,3952 6,8723 0,9520Table 2. Psycholinguistic parameters controlled to get equivalent datasets for each category of pictures in dataset 1 (Animals vs Tools).
Both predicted and experimentally derived coordinates of IDRs and annotations related to their function are stored in a variety of dedicated databases: DisProt  23  , MobiDB  24  , IDEAL  25  , DIBS  26  and MFIB  27  each focus on particular aspects of the ID spectrum. More recently, IDR annotations are also included in some core data resources including InterPro  28  , UniProt  29  and PDBe  30. Intrinsic structural disorder binding predictions are widely used, but an assessment of these predictors has never been systematically performed and is badly needed. In this report, we describe the first edition of CAID, a biennial experiment inspired by the critical assessment of protein structure prediction (CASP) for the benchmarking of ID and binding predictors on a community-curated dataset of 646 novel proteins obtained from DisProt  23.
Results CAID was organized as follows (Fig. 1a ). Participants submitted their implemented prediction software to the assessors and provided support to install and test them on the MobiDB servers. The assessors ran the packages and generated predictions for a set of proteins for which disorder annotations were not previously available. Given a protein sequence, the task of an ID predictor is to assign a score to each residue for its propensity of being intrinsically disordered at any stage of the protein's life. In CAID, we evaluated the accuracy of the prediction methods as well as software runtimes, which directly impact their suitability for large-scale analyses. Structural properties of proteins can be studied by a number of different experimental techniques, giving direct or indirect evidence of disorder.
Different techniques are biased in different ways. For example, IDRs inferred from missing residues in X-ray experiments are generally shorter because longer, noncrystallizable IDRs are either excised when preparing the construct or are detrimental to crystallization. At the other end of the spectrum is circular dichroism, which can detect the absence of fixed structure in the full protein but does not provide any information about IDR coordinates. IDR annotations are more reliable when confirmed by multiple lines of independent and different experimental evidence. In this first round of CAID, we selected the DisProt database as the reference for structural disorder because it provides a large number of manually curated disorder annotations at the protein level, with the majority of residues annotated with more than one experiment  23.
DisProt annotates IDRs of at least ten residues likely to be associated with a biological function and excludes short loops connecting secondary structure elements. DisProt also contains protein-protein interaction interfaces falling into disordered regions, used as a separate dataset (DisProt-binding). Ideally DisProt annotations would be complete-that is, each protein would be annotated with all disordered (or binding) regions present under physiological conditions. If this were true, we could simply consider all residues to be structured (that is negatives) when not annotated as disordered (that is, positives). Since not all IDRs are yet in DisProt, we created the DisProt-Protein Database (-PDB) dataset, where negatives are restricted to PDB Observed residues (Fig. 1c ). This dataset is more conservative but can be considered more reliable as it excludes 'uncertain' residues that have neither structural nor disorder annotation.
Compared to DisProt, DisProt-PDB is more similar to datasets used to train some disorder predictors (for example, refs. 19, 20, 22 ) and for CASP disorder challenges  31. The distribution of organisms reflects what is known from other studies  4, 5  , with the majority of ID targets coming from eukaryotes, a good representation of viruses and bacteria but much fewer from archaea (Fig. 1f ). At the species level, annotations are strongly biased in favor of model organisms with a majority from human, mouse, rat, Escherichia coli and several other common model organisms (Supplementary Fig. 6 ). Target proteins are not redundant at the sequence level, and are different from known examples available in the previous DisProt release. Mean sequence identity is 22. 2% against the previous DisProt release and 17.
IDR prediction performance The quality of IDR prediction can be evaluated in different ways. In some cases, it is relevant to know the fraction of disorder while in others it is more important to know the exact position of the IDR in the sequence. Since disorder can be used as a proxy either to estimate the complexity of an organism or complement a sequence search, it is also important for a predictor to be sufficiently rapid for genome-scale application. For CAID, we report the maximum F1-score (F max -that is, maximum harmonic mean between precision and recall across all thresholds), which takes into account predictions across the entire sensitivity spectrum  32. The performance of top methods, based on F max and calculated over all targets, is shown in Figs. 2  and 3  for the datasets DisProt and DisProt-PDB, respectively.
The F1-score, which is insensitive to dataset imbalance (Fig. 1d ), provides a ranking almost identical to that obtained with Matthews correlation coefficient (MCC). Supplementary Figs. 12, 13, 33 and 34 show a full comparison and the dependence of F1-score and MCC on predictor confidence scores, along with the predictor default confidence threshold (Supplementary Figs. 10,  11, 30 and 31). All methods were compared with the various baselines described in Methods. In some applications, the objective was to predict which protein fragments are disordered based on known examples in the PDB. This is a different problem than prediction of functional IDRs-for example, aiming to evaluate their biophysical properties. The naive baselines help us understand this difference and assess the effectiveness of the transfer-by-homology of structural information for IDR prediction (Discussion).
In the PDB Observed baseline, mimicking perfect knowledge, all residues not covered by any PDB structure are labeled as disordered. Alternatively, in the Gene3D baseline, residues are considered disordered if they do not match any Gene3D prediction for homologous domains. In the Shuffled dataset baseline, the reference is randomly shuffled at the dataset level while Random is an actual random predictor that does not use any previous knowledge. The values of F max (Fig. 2b, d ) and area under the receiver operating characteristic (ROC) curve (AUC) (Fig. 3e, g ) were substantially different when predictors were tested on the DisProt dataset, which contained uncertain residues, as opposed to the DisProt-PDB dataset. By definition, the PDB Observed baseline cannot predict negative residues outside PDB regions: it generates 56.
5% false positives, which dropped to zero when considering the DisProt-PDB dataset in which the uncertain residues are completely filtered out. IDRs overlapping PDB regions, usually corresponding to residues involved in folding-upon-binding events, instead generate false negatives. These are far less common (20. 4%) and remain the same for the two datasets. The Gene3D baseline typically increases PDB coverage (negatives), exploiting the transfer-by-homology principle. As a consequence, the probability of false positives is lower (48. 6%) and false negatives are only marginally more frequent (20. 9%). For the DisProt dataset, Gene3D slightly outperforms PDB Observed in terms of both F max (Fig. 2b, d ) and AUC (Fig. 3e, g ). Rather, for the DisProt-PDB dataset, PDB Observed is notably superior to all methods with only 6.
17, 18, 23, 28, 38, 39, 44 and 49), these methods can be seen broadly as performing consistently well. Looking at the precision-recall curves (Fig. 2d ), we notice that the top five methods (excluding fIDPnn/lr in the DisProt dataset and AUCpred-np in the DisProt-PDB dataset) leverage evolutionary information, introducing a database search as a preliminary step. The performance gain, on average 4. 5% in terms of F max , comes at the cost of slowing prediction by two to four orders of magnitude (Fig. 2c  and Supplementary Figs.
Fully disordered proteins We considered fully disordered proteins (IDPs) separately because these are particularly challenging to investigate experimentally; for example, they cannot be probed with X-ray crystallography yet they are of great interest because they fulfill unique biological functions  5, 33. We therefore designed another classification challenge: separation of IDPs from all other proteins. We consider proteins as IDPs when at least 95% of residues are predicted or annotated as disordered, and predictors were asked to identify IDPs based on this criterion. According to this definition, the number IDPs in the DisProt dataset is 40 out of 646. Different threshold values did not substantially affect the ranking (Supplementary Tables  6 7 8 ). In Table  1  all methods are sorted based on F1-score.
False positives are limited for many methods, although correct IDP predictions are generally made for less than half of the dataset. The fraction of residues predicted as disordered is also notably different across methods (Supplementary Fig. 50 ), suggesting room for improvement. Methods using secondary structure information may be at a disadvantage for IDP prediction, since annotations frequently rely on detection methods without residue-level resolution (for example, circular dichroism; Supplementary Fig. 7 ).
Prediction of disordered binding sites As a second major challenge, CAID evaluated the prediction of binding sites within IDRs, commonly referred to as linear interacting peptides  24  or short linear motifs  34  leveraging DisProt annotations for binding regions (Supplementary Fig. 52  shows dataset composition and overlap to other databases). In DisProt, binding annotations retrieved from the literature are fraught with more ambiguity than disorder examples. In addition, experimental evidence for the exact position of a binding region is often inaccurate because binding is annotated as a feature of an IDR. Our reference includes all entries in the DisProt dataset, even if they were not annotated with binding regions. This translates to a dataset where the majority of targets (414 out of 646) have no positives.
Software implementation We also evaluated those technical aspects related to software implementation-that is, speed and usability-that have a direct impact on their application for large-scale analyses. Speed in particular is highly variable, with methods of comparable performance varying by up to four orders of magnitude in execution time (Supplementary Fig. 4 ). In general, all methods incorporate a mix of different scripts and programming languages. Some software configuration scripts contain errors. In many cases data paths and file names are hardcoded in the program-for example, the sequence database or output file path. Only a few programs allow specification of a temporary folder, which is important for parallel execution. It is possible to provide precalculated sequence searches for only a few methods.
Some proteins remain unfolded until they bind a partner  40  while others are disordered providing they are in a specific cellular compartment and fold following translocation  41  , and some enzymes undergo order-to-disorder-to-order transition as part of their catalytic cycle  42. Given these challenges, CAID represents a community-based effort to develop and implement evaluation strategies to assess (1) clear definitions of ID and (2) the performance of methods used in the prediction of ID. In its first round, CAID leverages the DisProt database  23  of curated experimental evidence to assess ID predictors. In DisProt, curators store the coordinates of IDRs when there is experimental evidence in peer-reviewed articles of highly mobile residue stretches longer than ten residues. We anticipate that future rounds may include reference data arising from ever-improving consensus operational definitions-for example, NMR measurements, which are particularly powerful in the characterization of experimental protein disorder.
To overcome this problem, which is intrinsic to how we detect and store data, ID predictor performance was tested in two scenarios. In the first of these we assumed that all annotations were complete, considering all residues outside of annotated regions as structured. In the second scenario, we used resolved residues from PDBs to annotate structure and filtered out all residues that were covered by neither disorder nor structure annotation. Binding site predictors were tested on a dataset where all residues outside of binding regions are considered nonbinding. Despite these challenges, CAID revealed progress in the detection of ID from sequence and highlighted that there remains scope for improvement in both disorder and binding site predictors. One of the primary goals was to determine whether automated algorithms perform better than naive assumptions such as sequence conservation or three-dimensional structure.
As far as ID is concerned, the performance of predictors in comparison to naive baselines largely depends on the assumption made on nondisordered residues. On the DisProt-PDB dataset, where disorder is inferred from DisProt annotation, order from the presence of a PDB structure and all other residues is filtered out: naive baselines outperform predictors. However, when only DisProt annotations are considered (DisProt dataset), the tables are turned and predictors, while obtaining lower overall scores, outperform naive baselines (Figs. 2  and 3 ). When uncertain residues are retained in the analysis (DisProt dataset), the number of false positives increases and precision plunges, lowering the F1-score. This means that either predictors detect ID in the uncertain residues-suggesting that DisProt annotation is incomplete, predictors overpredict or both.
However, the absence of a protein from PDB may be simply due to the lack of studies on that protein. Gene3D performs better since it generalizes from existing structures, but still tends to overpredict disorder (or underpredict order). At the opposite side of the spectrum, methods that are too conservative in their disorder classification (for example, MobiDB-lite) perform NATuRE METhoDS | www. nature. com/naturemethods worse than expected on fully disordered proteins. Results from the DisProt dataset suggest that several methods are consistently among the top performers, although the exact ranking is subject to some variation. fIDPnn and SPOT-Disorder2 perform consistently well, with RawMSA and AUCpreD following closely. The execution times for these four methods vary by up to three orders of magnitude, suggesting there is room for optimization of the software.
Furthermore, while it is arguable that this evaluation has limitations, its publication helps highlight such constraints and exposes this problem to the rest of the scientific community. We compared predictors to the same baselines used for the disorder challenge but, while their design remains unchanged, their underlying naive assumption changes slightly. The PDB Observed baseline assumes that whatever is not covered by a structural annotation in PDB is not only disordered but also involved in one or more interactions. When considering all targets in the CAID dataset, including those not annotated as binders, predictors slightly outperform the baselines but have limited performance overall. Figure  4  shows disagreement with the DisProt-binding reference in both positive and negative classification, highlighting the potential for improvement of binding predictors.
We have to consider that the dataset used is strongly unbalanced. Although a prominent function of IDPs is mediation of protein-protein interactions, most targets (414 of 646) do not contain an identified binding region and those that do include binding regions often have them spanning the whole disordered region in which they are found. This strong bias is due to how DisProt was previously annotated, with the label 'binding' being associated with an entire IDR. In the latest DisProt version this annotation style has been replaced with a more detailed one, ensuring that future editions of CAID will be less biased towards long binding regions. The improved definition of boundaries for disordered binding regions could favor methods trained specifically to recognize shorter binding regions. Overall, this suggests a large growth potential in both predictors and reference sets for this challenge.
Acknowledgements Development of the predictors was supported in part by the  National Science Foundation  (grant no. 1617369 ),  Natural Sciences and Engineering Research Council of Canada  (grant no. 298328 ),  Tianjin Municipal Science and Technology Commission  (grant no. 13ZCZDGX01099 ),  National Natural Science Foundation of China  (grant nos. 31970649  and  11701296 ),  Natural Science Foundation of Tianjin  (grant no. 18JCYBJC24900 ),.
Data availability Raw DisProt annotations, reference datasets and predictions in CAID format are available at https://idpcentral. org/caid/data/1/. Description of the process and code to produce references is available in the GitHub CAID repository at https://github. com/BioComputingUP/CAID. All data used in the analysis are also available in the Code Ocean capsule (https://doi. org/10. 24433/CO. 3610625. v1).
Code availability Results of the CAID assessment can be fully reproduced by downloading the code and following the instructions in the CAID GitHub repository at https:// github. com/BioComputingUP/CAID. The CAID software is a Python 3 package that produces all outputs necessary for CAID, including baselines, references and plots. See Data availability for information about how to obtain Input predictions, references and sequence annotations. The CAID package is dependent on public Python 3 libraries and on the vectorized_cls_metrics library, available at https:// github. com/marnec/vectorized_cls_metrics. The code is also available and reproducible in the Code Ocean capsule (https://doi. org/10. 24433/CO. 3610625. v1).
Methods All software programs were executed using a homogeneous cluster of nodes running Ubuntu 16. 04 on Intel 8 core processors with 16 GB of RAM and a mechanical hard disk. In the text we refer to proteins as targets, to disordered residues as positive labels and to structured/ordered residues as negative labels. Experiment design is described in the Nature Research Reporting Summary. Reference sets. In CAID different reference sets were built, differing in the subset of DisProt used to define positive labels and in the definition of negatives labels. For the disorder challenge, we generated two reference sets called DisProt and DisProt-PDB. Both references are composed of a set of 646 targets, annotated between June 2018 and November 2018 (DisProt release 2018_11). Positive labels in both reference sets are those residues annotated as disordered in the DisProt database.