dataesr/scientific-paragraphs-categorization huggingface dataset:
This is one of the major possible effects of time pressure as discussed here, and the one that has been least studied in the context of intelligence tests. Third, the experiment only examined the effects of time pressure at the task level, not at the item level (e. g. ,  Kyllonen et al. 2018). item-level time pressure in the APM is less straightforward, because items of different difficulties have very different RTs (see Table  4 , Figure  4 ). This calls for different time limits, given that applying the same moderate time limit to all items could result in easy items becoming practically unspeeded and difficult items becoming practically unsolvable. An experiment with variable time pressure at the item level could be interesting, although a preexisting dataset (such as the one provided here) would be necessary to calibrate appropriate time limits.
Using genome-wide ortholog sets and taking advantage of the high-quality human and Saccharomyces cerevisiae genome annotations, we analyzed the domain gain and loss events in the predicted proteomes of 9 non-human primates (NHP) and 20 non-S. cerevisiae fungi (NSF) as annotated in the Uniprot and Interpro databases. Our approach allowed us to quantify the impact of errors on estimates of protein domain gains and losses, and we show that domain losses are over-estimated tenfold and three-fold in the NHP and NSF proteins respectively. This is in line with previous studies of gene-level losses, where issues with genome sequencing or gene annotation led to genes being falsely inferred as absent. In addition, we show that insistent protein domain annotations are a major factor contributing to the false events.
For the first time, to our knowledge, we show that domain gains are also over-estimated by three-fold and two-fold respectively in NHP and NSF proteins. Based on our more accurate estimates, we infer that true domain losses and gains in NHP with respect to humans are observed at similar rates, while domain gains in the more divergent NSF are observed twice as frequently as domain losses with respect to S. cerevisiae. This study highlights the need to critically examine the scientific validity of protein annotations, and represents a significant step toward scalable computational fact-checking methods that may 1 day mitigate the propagation of wrong information in protein databases.
, 2019; Yu et al. , 2019; Nevers et al. , 2020). Gene-level methods are limited especially when comparing complex multi-domain proteins or phylogenetically distant species. To overcome these bottlenecks, protein domains have been suggested as an alternative unit for studying groups of functionally equivalent proteins  (Gabaldon and Koonin, 2013; Syamaladevi et al. , 2013; Wang et al. , 2021). Differences in protein domain composition can be common among orthologs and are functionally important, especially for multicellular eukaryotes where domain architectures are often highly complex and lineage specific  (Sjolander et al. , 2011; Forslund and Sonnhammer, 2012; Persson et al. , 2019). Underscoring their importance, ortholog databases are beginning to include domain level information  (Forslund et al.
, 2018; Nevers et al. , 2022). The most common changes observed in both ortholog and paralog pairs involved loss/gain of domains, while domain shuffling and segment duplication/deletion are less frequent  (Forslund et al. , 2011). However, there has been some debate about the true level of domain conservation of orthologs  (Lin et al. , 2006; Forslund et al. , 2011; Nagy et al. , 2011; Dohmen et al. , 2020). Notably, a comparison of domain architectures for orthologous proteins in the high quality manually curated UniProtKB/Swiss-Prot database  (Uniprot Consortium, 2022)  suggested that the rate of architecture alteration is very low: the domain architecture of ~5% of the orthologs is changed over 1,000 My  (Nagy et al. , 2011). In contrast, comparison of UniProtKB/TrEMBL or RefSeq predicted protein sequences identified a significantly higher proportion of differences, for example 4.
01% of human-mouse orthologs in TrEMBL had different architectures compared to 1. 1% in SwissProt. The explanation for this difference is that the predicted sequence databases are significantly contaminated with incomplete or chimeric protein sequences that have missing domain annotations compared to their complete Swiss-Prot orthologs. Since then, a growing body of evidence has revealed significant levels of other types of protein annotation errors in public databases  (Schnoes et al. , 2009; Steinegger and Salzberg, 2020; Rembeza and Engqvist, 2021; Goudey et al. , 2022). The starting point for the complex process of protein domain annotation is the availability of high quality genomes. For technical and methodological reasons, available genome sequences are of different quality: some are complete, while others are incomplete or in draft state (http://www.
genomesonline. org/cgi-bin/GOLD/ index. cgi). The subsequent genome annotations of gene intron/ exon structures are also of highly diverse quality. For model genomes, for example from human, mouse or the yeast Saccharomyces cerevisiae, extensive transcriptome data is available allowing more accurate gene identification  (Weirather et al. , 2017). For other organisms, the genomes are automatically annotated and gene/protein sequences have not been fully verified. This is particularly serious for draft genomes, misleading on the actual number of genes/isoforms  (Deutekom et al. , 2019; Weisman et al. , 2022)  and the corresponding protein sequences  (Bányai and Patthy, 2016; Tørresen et al. , 2019; Meyer et al. , 2020). Finally, it has been shown that default protein domain annotations, generally based on searches for known protein domains listed in public databases like  Interpro (Blum et al.
, 2020). We apply the method to two genome-scale ortholog sets containing the predicted proteomes of 9 non-human primates (NHP) and 23 non-S. cerevisiae fungi (NSF) respectively from the UniprotKB database, and evaluate all potential domain gain and loss events using the wellstudied human and S. cerevisiae proteomes as a reference. However, the method should be easily applicable to other ortholog sets, where it can be used to filter the potential domain events that are due to annotation errors, thus highlighting candidates for true evolutionary novelty.
Protein sequence data Human and S. cerevisiae proteins were identified in the Uniprot database version 2022_03. For each protein, one-to-one orthologs for 9 NHP and 21 NSF species were then extracted from the OrthoInspector database (v3)  (Nevers et al. , 2019). For all proteins, the canonical sequences were retrieved from Uniprot and domain architectures were retrieved from the Interpro database (Pfam-A only) using the API in json format. Protein sequences longer than 5,000 amino acids were eliminated for convenience. The corresponding genomic sequences were then retrieved from the Ensembl database  (Martin et al. , 2022) , using the pre-defined gene positions extended by + -1,000 nucleotides. Genome level information was also obtained from Ensembl, including the total number of coding genes and N50 statistics.
Identification of inconsistent domain annotations For all ortholog pairs with different domain contents, the domain annotations were re-computed using the same protocol. Pfam HMM-profiles (version 35. 0 November 2021) were downloaded from http://ftp. ebi. ac. uk/pub/databases/Pfam/ current_release/Pfam-A. hmm. gz and installed locally. Then, HMMSEARCH (version 3. 3. 2) was used to search each protein sequence against the Pfam HMM-profiles with an E-value threshold of 0. 01 and the gathering (GA) threshold option.
Classification of Pfam-A domains into clans The domain entries in Pfam are grouped hierarchically into clans if they are considered to have arisen from a single evolutionary origin, assessed by the presence of related 3D structures, related functions, significant matching of the same sequence to HMMs from different families and profile-profile comparisons. We retrieved the clan classification for all Pfam-A domain entries from the Interpro resource at https://ftp. ebi. ac. uk/pub/databases/Pfam/current_release/ Pfam-A. clans. tsv. gz.
Phylogenetically corrected regression analysis (PGLS) To estimate the phylogenetic distance between the reference species (human/S. cerevisiae) and non-reference species (NHP/ NSF), (i) for NHP, we downloaded the species tree from the Ensembl database, and (ii) for NSF, we retrieved a recently published genome-level phylogeny  (Li et al. , 2021). The trees were then used to calculate patristic distances (i. e. sum of branch lengths) between species. Finally, to test for correlations between the number of domain events observed and the phylogenetic distance, we used the phylogenetic generalized least squares (PGLS) method implemented in the CAPER (http://cran. rproject. org/package=caper) v0. 5 package for R. Following standard practice, delta (δ) and kappa (κ) were set to 1 while Fact checking protocol for potential missing domain (MD) and additional domain (AD) events in a given ortholog from a non-reference species with respect to the protein from the reference species.
Verification of annotated protein domains For both MD and AD, the first step is the identification of inconsistent protein domain annotations. The domain annotations for all reference and non-reference proteins were extracted from the Interpro database, using only Pfam-A domains. The annotations in the Pfam database are constructed using a combination of human expert curation and automatic predictions from statistical models. InterPro also integrates predictive models from several other databases, and does not apply a single E-value threshold applied to all entries. This generally improves accuracy, but can lead to inconsistent annotations between organisms (). To identify inconsistent domain annotations, we reannotated the proteins that had different domain contents using a standardized protocol based on hmmsearch queries against a locally installed version of the PFAM-HMM domain profiles, and a defined E-value threshold of 0.
Verification of gene/isoform predictions For potential MD events in the non-reference orthologs, the second step involves verification of the gene/isoform prediction and the inferred protein sequence. We extracted the genomic sequence coding for the non-reference protein from the Ensembl database, extending the gene sequence 1,000 nucleotides upstream and downstream of the defined gene region. We then used TBLASTN to search for a genome sequence region potentially coding for the missing domain. For potential AD events in the ortholog proteins, the gene prediction verification was unnecessary, since the domain is present in the non-reference ortholog, but absent from the reference protein. Gene predictions in these well-studied genomes were considered to be correct. However, it is possible that there are inconsistencies in the definition of the canonical isoforms in Uniprot, and that the canonical sequence defined in the nonreference organism is orthologous to an alternatively spliced reference protein.
Construction of genome-scale data sets We applied our protocol to two data sets containing: (i) 9 NHP proteomes (Table  1 ) which were compared to the reference human proteome, (ii) 20 NSF proteomes (Table  2 ) compared to the reference S. cerevisiae proteome. All protein sequences were extracted from the UniprotKB database. The two datasets represent some of the main challenges for domain annotation pipelines. First, the NHP species are closely related (with proteomes sharing >89% sequence identity with human), but proteins are generally more complex, with multi-domain architectures. Furthermore, some of the genome assemblies are incomplete, as estimated by their contig N50 metric of <100 Kb compared to >10 Mb for the higher quality genomes or 55 Mb for the human genome. The contig N50 is defined as the length of the shortest contig for which longer and equal length contigs cover at least 50% of the assembly.
The NSF represent more divergent species (mean sequence identity ranging from 24. 5% to 41. 8% with S. cerevisiae). However, the genomes are smaller and gene prediction is simpler since the genes are less intron-rich. We then identified all ortholog pairs between each reference proteome (human, S. cerevisiae) and the corresponding nonreference proteomes (NHP, NSF respectively). Orthologs were extracted from the Ortholnspector database and the sequences that were cross-referenced in the Ensembl genome database were retained. In order to avoid redundancy and bias in the data sets, only one-to-one orthologs were considered. For the primate dataset, this resulted in 19,251 human proteins (out of 19,813) that had at least one-to-one ortholog, and a total of 152,240 orthologous NHP sequences (Table  1 ).
Of the 152,240 orthologous sequences, 2,264 (1. 5%) were extracted from the SwissProt database, and the remaining 149,976 (98. 5%) were from TrEMBL. The number of orthologous NHP sequences retained is generally similar for each species, ranging from 16,071 to 17,944. For the fungi dataset, 3404 S. cerevisiae proteins (out of 6,600) were retained and a total of 40,410 orthologous NSF sequences (Table  2 ). Of the 40,410 orthologous sequences, 4,853 (13. 6%) were extracted from SwissProt, and 355,557 (86. 4%) were from TrEMBL. The proportion of S. cerevisiae proteins with one-to-one orthologs is smaller (52%) compared to human proteins (97%), due to the higher divergence of the selected species in the fungi dataset. The number of orthologous NSF sequences retained for each species ranges from 1819-2,889, with the exception of the two obligate unicellular parasites (Nosema ceranae and V.
Identification of all potential MD and AD events For all proteins, we retrieved their Pfam-A domain annotations from the Interpro database and compared the domain contents of all ortholog pairs, where an ortholog pair consists of a protein from the reference species and its one-to-one ortholog in a non-reference species. For the primate dataset (Table  3 ), a total of 5,695 (3. 6%) of the 152,240 orthologous NHP sequences had different domain contents, with 4,462 orthologs presenting a potential missing domain (MD) with respect to human and 1,233 orthologs presenting a potential additional domain (AD). Interestingly, only 2 (0. 04%) of the 4462 MD and 4 (0. 3%) of the 1233 AD were found in sequences from the SwissProt database. The number of potential MD varied from 237 for chimpanzee to more than 700 for gibbon and vervet.
The number of potential AD was smaller for all NHP species, ranging from 92 to 199, and leading to a mean MD/AD ratio of 3. 6. The MD/AD ratio varied significantly from 1. 9 for macaque to 6. 1 for vervet. For the fungi dataset (Table  4 ), a total of 7772 of the 40,410 orthologous sequences had different domain contents, i. e. 17. 6%, with 3987 orthologs (including 233 or 6. 2% from SwissProt, and 3754 or 93. 8% from TrEMBL) presenting a MD with respect to S. cerevisiae and 3785 orthologs (including 289 or 7. 6% from SwissProt, and 3496 or 92. 4% from TrEMBL) presenting an AD. The Microsporidia parasites N. ceranae and V. culicis had a smaller number of potential AD (15 and 14 respectively) as might be expected since they are obligate unicellular parasites with a smaller proteome (467 and 491 protein orthologs respectively), but a much larger number of  MD (178 and 191 respectively).
Verification of potential missing domain (MD) events We used our fact checking approach (Figure  2 ) to evaluate all non-reference (NHP/NSF) proteins that had potential MD events according to the Pfam-A domain annotations in the Interpro database. First, for the 4462 NHP orthologs and 3987 NSF orthologs with potential MD compared to the reference (human/S. cerevisiae), we identified 1713 (38%) and 2008 (50%) MD events for NHP and NSF respectively linked to inconsistent domain annotations. These false positive MD events include 1,147 and 1,614 cases for NHP and NSF respectively, where the MD was identified in the orthologous sequences by our protocol. In other words, while the NHP/NSF protein was not annotated with the domain in the Interpro database, increasing the hmmsearch E-value allowed us to identify the MD.
As an illustrative example, the human protein P55318 (FOXA3_ HUMAN: transcription factor forkhead box protein 3) and the macaque ortholog F6Q3J5 both contain a central forkhead domain (PF00250), but the forkhead N-terminal region (PF08430) is not annotated in the macaque sequence. The N-terminal region was found in the human sequence with an hmmsearch E-value = 3. 9 × 10 -10 , and in the macaque sequence with E-value = 1. 6 × 10 -8. This domain is also identified by a new Deep Learning method  (Bileschi et al. , 2022)  that was trained on Pfam data and annotated as a Pfam-N (N for Network) domain in Interpro. More surprisingly, we discovered 566 and 394 cases for NHP and NSF respectively, where the MD was not detected in the reference sequence by our protocol. Thus, the protein is annotated with the domain by Interpro, but is not detected by hmmsearch with an E-value threshold of 0.
01. As an example, the S. cerevisiae protein Q05580 (HEL2_YEAST) is annotated with the Interpro domain "Zinc finger C2H2-type" (IPR013087), which combines signatures from two member databases SMART (SM00355) and Pfam (PF00096). The annotation file retrieved from Interpro contains both signatures, even though the Pfam signature does not match this specific protein with an E-value below the threshold. Next, we focused on the identification of gene/isoform prediction errors for the remaining 2749 NHP and 1979 NSF potential MD events that could not be explained by domain annotation issues. After applying a simple correction protocol based on alignment of the reference protein sequence with the genomic sequence of the ortholog (see Methods), the MD was identified in 1368 NHP and 464 NSF orthologs.
An illustrative example is shown in Figure  3  for the potential MD event in the bushbaby sequence B5FW94, with respect to the human sequence Q15904 (VAS1_HUMAN). VAS1_HUMAN is described as an accessory subunit of the proton-transporting vacuolar (V)-ATPase protein pump, which is essential in supporting intracellular membrane trafficking and protein degradation, which in turn are important for immune responses, cell signaling, and neurotransmitter release  (Wang et al. , 2020). The human protein contains a luminal domain (PF05827), as well as a C-terminal transmembrane domain (PF02274) that is potentially lost in the bushbaby ortholog B5FW94. However, a search of the corresponding bushbaby genome identifies a sequence segment downstream of the gene that could represent an additional exon coding for the MD.
Both proteins are annotated in Interpro as belonging to the GT29-like superfamily (IPR038578), but CT173_ HUMAN has no Pfam annotations. H2P1R9_PONAB is annotated with a glycosyltransferase family 29 domain (PF00777) in the Interpro database and the domain is also identified using hmmsearch with an E-value = 2. 0e-10. Using hmmsearch, PF00777 is also found in the CT173_HUMAN sequence with E-value = 3. 2e-08. After the first step, 529 NHP AD events and 2161 NSF AD events remained as potentially true domain events. In the second step, we searched for alternative isoforms in the reference species (human/S. cerevisiae) in the Ensembl database, and checked whether an isoform was present that contained the AD. This step is only pertinent for the primate data set, since none of the S. cerevisiae genes corresponding to the potential AD events had known isoforms.
Origin of errors leading to false MD and AD events Our evaluations of the potential MD and AD revealed that a significant proportion of the events can be attributed to systematic errors. Overall, inconsistent domain annotations accounted for the most errors (NHP: 42. 7%, NSF: 46. 7%). Gene prediction or isoform errors were the second most frequent causes of false positive domain events (NHP: 26. 1%, NSF: 6. 0%). To further investigate the causes of the false domain events, we compared the rates of MD and AD events observed before and after the fact-checking protocol with two measures of genome and gene annotation quality. It should be noted that we excluded the two microsporidia species N. ceranae and V. culicis from the NSF dataset, due to the bias in the MD/AD ratio for these parasitic fungi.
First, we used the contig N50 as an estimate of genome sequence/assembly completeness (from Table  1 ) and compared the N50 values to the frequency of domain events using graphical plots (Supplementary Figure  S1 ) and Spearman rank correlations (Supplementary Table  S1 ). For both NHP and NSF datasets, the correlation between N50 and the number of MD events is reduced after fact-checking (for NHP: potential MD R = -0. 7, p = 0. 04, true Frontiers in Bioinformatics frontiersin. org MD R = 0. 2, p = 0. 55; and for NSF: potential MD R = -0. 4, p = 0. 08, true MD R = -0. 02, p = 0. 93), suggesting that the fact-checking protocol was successful in removing this bias. However, the correlation between N50 and the number of AD events is more complex. For NHP, the correlation is increased for true AD events (R = 0.
23, p = 0. 04). Thus, our fact-checking protocol generally increases the correlation between phylogenetic distance and the number of predicted events, at least for MD, supporting the hypothesis that true MD events occur more frequently in more distantly related species. Again, the correlation for the number of AD events is more complex, and potential reasons for this are discussed below. The proportion of potential MD events identified as true events is different for the two datasets: 9. 2% for the NHP proteomes, and 31. 2% for the NSF proteomes. The proportion of true AD events is also different: 32. 2% for the NHP proteomes, 57. 1% for the NSF proteomes. The larger proportions of true AD compared to true MD for both NHP and NSF are possibly due to protocol issues since false positive MD can be more easily identified by checking for their presence in the corresponding genome sequence.
Impact of errors on estimates of MD and AD events The error detection protocol presented in the previous section allowed to filter false positive predictions and to more reliably estimate the rates of MD and AD events in the NHP/NSF proteomes (Figure  6 ). After filtering, the mean number of NHP orthologs with different domain contents is reduced from 3. 7% to 0. 5%. This is in line with previous observations  (Nagy et al. , 2011) , for example for human-orangutan comparisons, a rate of 0. 3% orthologs with different domain architectures was identified in expert-reviewed proteins from the SwissProt database. Filtering of the NSF dataset also resulted in a reduction of the number of orthologs with different domain contents from 17. 6% to 7. 7%, although the proportion of true events is larger than for NHP proteins.
For the NHP dataset, the potential MD events inferred from the public database annotations are more frequent than AD events (2. 9% of orthologs with MD and 0. 8% of orthologs with AD: paired t-test, t (10) = -5. 70, p = 0. 0005). However, we conclude that this difference is due to annotation errors, since after filtering of the false positives, MD and AD rates are the same (0. 27% of orthologs with MD and 0. 26% of orthologs with AD: paired t-test t (10) = -0. 10, p = 0. 92). Similar rates of MD and AD might be expected, since the absence/presence of a domain was arbitrarily defined to refer to the NHP protein compared to the human reference. In contrast, for the NSF dataset, while the number of potential MD and AD events inferred from the public database annotations are similar (9. 1% of orthologs with MD and 8.