dataesr/scientific-paragraphs-categorization huggingface dataset:
The first set of three hypotheses focuses on whether error variance relates to various quality measures: H1 Team quality does not explain the size of errors in Stage 1 results. Team quality is proxied by the largest common factor in various candidate proxies for team quality. We prefer an appropriately weighted average over simply adding all proxies to maximize statistical power in the regressions. More specifically, we define team quality as the first principal component of standardized series that include: 18   (c) Experience with big data: The team has worked with datasets that are at least as big as the Deutsche Börse dataset analyzed in #fincap (0/1). (d) Academic seniority: At least one team member holds an associate or a full professorship (0/1). (e) Team size: The team size attains its maximum of two members (0/1).
Alternative explanations After having presented all our results, it is useful to discuss alternative explanations. Might the sizeable non-standard errors be due to the presence of inexperienced researchers testing unsuitable hypotheses with little effort? We believe this is unlikely to be the case for the following reasons. Experience. Aware of this potential pitfall, we selectively approached research teams and peer evaluators who we knew were sufficiently experienced in the field. When signing up, they ticked a box that they understood that participating in #fincap requires research expertise and experience in empirical finance and the analysis of large datasets. Ticking the box further meant that they acknowledge that one of the team members held a PhD in finance or economics. After ticking the box, researchers had to motivate in open text box why they believe they meet these requirements.
B Explanatory variables for error variance B. 1 Team quality The quality measures for research teams are based on the survey that participants filled out upon registration (see Figure OA. 2 in the Online Appendix). To keep the regression model both concise and meaningful, we reduce the ordinal variable "current position" and the logarithmic interval-based variable "size of largest dataset worked with" to binary variables. The academic position variable is one if a researcher is either associate or full professor. The dataset variable is one if the researcher has worked with datasets that are contained at least 100 million observations, because the #fincap sample contains 720 million observations. We aggregate these binary variables to research team level by taking the maximum across the team members.
The trial TE can be retrieved in two ways, based on either AD or IPD. In AD, the most common, researchers only retrieve the TE and its precision for each study. This approach is easier and faster to perform because researchers "only" have to retrieve the information published in the literature (article, abstract,. ). In IPD, researchers retrieve the full dataset of each trial, including covariates for each patient, and the TE, along with its precision, is recalculated. The latter is a fastidious work because of the difficulty to effectively retrieve individual information and laws regarding IPD are becoming more complex. Yet, IPD is considered the gold standard as it allows more control on the data to update follow-up and evaluate the heterogeneity of TE by including covariate-treatment interactions  [5].
Mixed-effects Cox's proportional hazard model (IPD-CoxME). A multilevel Cox proportional hazard model (CoxME) can be used for IPD NMA  [21, 22]. However, this approach leads to extensive calculation times when the complexity of the network and the number of random effects increase. BIn order to reduce computation time in the simulation study, this model was not used in the simulation study, but only in the MARCH/MACH-NC NMA data application.
ESE was defined as the SD of the estimated parameter over the 1,000 replicates. ASE was defined as the average of the standard errors estimated by each model  [27, 28]. All simulations were done with the R 3. 5. 2 software (The R Core Team, Vienna Austria, 2020) with the furrr 1. 14. 0 (Vaughan, 2018) packages for multicore computing and a L'Ecuyer-CMRG random number generator to ensure that each worker started at a different seed. Cox model was estimated by the coxme package 2. 2-16, multilevel Poisson's with lme4 1. 1. 21 [29] , contrast based-AD with Netmeta 1. 1. 0 and meta-regression with metaphor  [30]. The R code for all simulations is available on GitHub (https://github. com/Oncostat/One_step_ frequentitst_IPD_NMA).
In scenario 2 (in which the modifier was evenly distributed), conditional TEs at mean age were not biased. In scenario 3 (in which the modifier did not have the same distribution in all comparisons), the conditional TE at mean age was biased in AD-based models. Similar findings were observed on real data. One-step IPD-based Cox and Poisson models led to remarkably similar estimates and found no statistically significant differences between the indirectly compared treatments. On the other hand, an AD-based model, ignoring age, would have concluded that mRT was superior to chemoradiotherapy at a 0. 05 significance level. Meta-regression performed well in this (highly powered) datasets and correctly estimated the TE with a non-significant difference. Yet, this dataset illustrated an advantage of the onestep model and an issue of meta-regression: the ecological bias that led to the overestimation of the interaction magnitude.
Decomposition of the interaction in IPD-based models, as proposed by Hua et al. [8]  revealed that the within-trial interaction (estimated by standard IPD models) was much lower than the between-trial interaction (estimated by meta-regression). Our study also demonstrates the possibility to use either the Cox or Poisson model in a frequentist paradigm, both of them leading to similar results with our dataset. All models were successfully applied to the real data, including 79 trials and more than 18,000 patients, without convergence issues. IPD-based models were not too computationally intensive and took, on a standard desktop computer, 400 seconds for the mixed-effects Cox model and 100 seconds for the mixed-effects Poisson model. These figures are reasonable for a single application, notably when compared to Bayesian approaches.
When inconsistency is detected, adjusting for the confounders and/or adding an interaction term may mitigate the issue  [11, 12, 15]. In a recent work, Donegan et al. demonstrated that this problem is more subtle, as it involves both the intercept and slope of the covariate to TE relation  [13]. One-step IPD-based models allow flexible handling of this situation with comparison-specific intercept and interaction. Several strengths of our study should be highlighted. Simulations were done according to 48 scenarios representing a wide set of possibilities and two types of coding (continuous or qualitative) were used for age. Several statistical models were used, including two IPD one-step propositions. Finally, we observed on our real data application that our findings are not just theoretical but can lead to erroneous conclusions in the real world.
The amount of public proteomics data is rapidly increasing but there is no standardized format to describe the sample metadata and their relationship with the dataset files in a way that fully supports their understanding or reanalysis. Here we propose to develop the transcriptomics data format MAGE-TAB into a standard representation for proteomics sample metadata. We implement MAGE-TAB-Proteomics in a crowdsourcing project to manually curate over 200 public datasets. We also describe tools and libraries to validate and submit sample metadata-related information to the PRIDE repository. We expect that these developments will improve the reproducibility and facilitate the reanalysis and integration of public proteomics datasets. T he amount of proteomics data in public repositories is growing at an unprecedented rate 1,2.
ProteomeXchange (PX) is a consortium of proteomics resources, including the PRIDE database 2 , PASSEL and PeptideAtlas 3 , MassIVE 4 , jPOST 5,6 , iProX 7 , and Panorama Public 8. As of July 2021, over 27,000 datasets have been submitted to PX data repositories. PX datasets cover the whole spectrum of protein mass spectrometry (MS) analytical methods and experimental designs, which enable biologists and clinicians to study different aspects of the proteome. In parallel to the generalization of data deposition, reuse of public datasets is becoming increasingly popular. However, thus far, data reuse has largely been limited to.
benchmarking studies and applications related to peptide and protein identification, with resources such as PeptideAtlas  3  and GPMDB  9  systematically reanalyzing data from PX  10. Recently, new efforts like ProteomicsDB  11  , MassiVE. Quant  4  , and Expression Atlas  12  have started to include reanalyzed quantitative public datasets to present baseline and differential protein expression. However, the scalability and broad reuse of public quantitative experiments have been limited by the lack of sample metadata annotation, which unambiguously associates the samples included in each dataset with the corresponding data files  13, 14. Since 2012, PX resources have been capturing a general dataset description, including the dataset title, description, instrument, protein modifications included in the search, and submitters/ principal investigators, among other data  1.
The files included in each dataset are, on one hand, the output of the corresponding instrument (e. g. , RAW files), and on the other hand, the processed results, which can be represented, e. g. , in standard file formats such as mzIdentML  15  or mzTab  16. Currently, all PX partners mandate two types of information for each dataset: a general dataset description and the files containing the different required data types. Unfortunately, the experimental design and sample-related information are frequently missing in the datasets or are stored in ad hoc ways and/or formats  1. Information about the biological samples such as the analyzed organ, tissue, disease, or cell line, and the links between the samples and the corresponding data files are often lacking. Sample-related metadata and their relationship with the data files are well captured in two widespread file formats called ISA-TAB  17  and MAGE-TAB (MicroArray Gene Expression Tabular)  18  , which are used in metabolomics and transcriptomics, respectively.
As of May 2021, ArrayExpress has stored over 74,000 functional genomics datasets in the MAGE-TAB format  18, 19. In both formats, a tab-delimited file is used to annotate the sample metadata and link the metadata to the corresponding data files. While MAGE-TAB was originally designed for microarray experiments, it has been successfully adapted to high-throughput RNA-sequencing and single-cell RNA-Seq experiments  20. Here we introduce an extension and implementation of the MAGE-TAB format for proteomics (MAGE-TAB-Proteomics). The format has been developed in collaboration with the Proteomics Standards Initiative (PSI), the organization in charge of developing open-standard formats in the field  21. We have also developed general guidelines about what information needs to be encoded in MAGE-TAB to improve the reproducibility and enable the reanalysis of proteomics datasets.
In addition, we have crowdsourced the annotation of over 200 existing public datasets according to these guidelines, covering different analytical methods and experimental designs. Finally, we have developed an ecosystem of tools to validate MAGE-TAB-Proteomics files and integrate the metadata in the PRIDE database, the most popular PX resource. The full specification document describing all aspects of MAGE-TAB-Proteomics version 1. 0, the current implementations, as well as application examples, is available at the PSI website (https://psidev. info/magetab). Repurposing MAGE-TAB for proteomics. MAGE-TAB encodes the sample metadata annotations and the information linking the metadata to the corresponding data files in two different files: the Investigation Description Format (IDF) and the Sample and Data Relationship Format (SDRF).
In the following, we describe how we adapted these formats to the specific needs of proteomics. Providing study-description information in IDF. The IDF file contains information describing the study, including, e. g. , authors/submitters, protocols, and publications (Supplementary Note 1). The IDF format contains a series of key/value pairs, where each key represents a different property. For example, "Experiment Description" should be followed by a free-text description of the experiment (which would be the value). Most of the fields can contain more than one value, so that multiple values (e. g. , multiple-analysis software tools) can be defined in a single IDF file. Since 2012, PX dataset descriptions are provided using PX XML (http://proteomecentral. proteomexchange. org/ schemas/proteomeXchange-1.
4. 0. html), an XML file format that captures equivalent information to the ones included in IDF, making both files easily exchangeable (Supplementary Note 1). Therefore, we developed the IDF component of MAGE-TAB based on the existing PX XML format. Linking samples to data files with SDRF. SDRF is a tabdelimited file that describes the samples and allows their mapping to the data files 1. As shown in Fig. 1a , SDRF includes the annotation of (i) biological sample metadata; (ii) the relationships between samples and data files; (iii) (technical) metadata of RAW data files; and (iv) the variables under study (called factor values). Each row in an SDRF file corresponds to one relationship between a sample and a data file (an MS RAW file or a channel included in a given RAW file in the case of labeling-based proteomics).
Each sample in an SDRF file has a unique identifier (source name), and every sample property is encoded using the prefix characteristics (e. g. , characteristics [organism part]). Each data file also has a unique identifier (assay name), and every file property has the prefix comment (e. g. , comment[instrument], comment[fraction identifier]). Finally, the variables under study must be specified with the prefix factor value (e. g. , factor value[tissue]). The MAGE-TAB-Proteomics specification defines the minimum information that should be provided for every sample and data file (https:// github. com/bigbio/proteomics-metadata-standard/raw/master/psidocument/HUPO-PSI-MAGETAB-Proteomics_latest. docx). For all proteomics experiments, the following properties must be provided: organism, organism part, and biological replicate accession.
For every data file, the following properties are required: fraction identifier, technical replicate accession, label (in the case of labeling methods), and data-file name. Biological and technical replicates should be explicitly included using the terms characteristics[biological replicate] and comment[technical replicate], respectively (Fig. 1a ). The biological replicate field is considered a property of the samples, whereas the technical replicate is considered a property of the data files. A second category of fields includes information that is not mandatory but recommended. Each PX repository can define which of the recommended fields must be provided in their resource, depending on the experiment types. The current PX templates request the submitters to provide the following properties for every data file: instrument model, cleavage agent, fragment-mass tolerance, precursor-mass tolerance, and mass modifications (e.
g. , post-translational modifications (PTMs) and artifactual modifications considered in the analysis). Most of the values of these properties can be encoded as a combination of multiple key/value pairs (e. g. , methionine oxidation can be specified as AC = UNIMOD:35;NT = Oxidation;MT = Variable;TA = M). We believe that this represents the minimum set of information that is necessary for practical terms for understanding and re-using MS-based proteomics datasets. Furthermore, other properties such as labeling can be provided. Importantly, the set of mandatory properties can be readily expanded in case the PX community decides to extend its metadata requirements in the future. For now, we have defined additional templates (Supplementary Table  1 ), which form a set of recommended properties required per experiment type.
Submitters can use the template corresponding to their experiment to streamline annotation. For example, the cell line (characteristics[cell line]) is a recommended metadata item for cell-line experiments; for every human dataset, the disease under study should be provided in the field characteristics[disease] and the control samples should be labeled with the value "normal". Multiplexing and fractionation. Transcriptomics datasets typically show a one-to-one relationship between each sample and data file. While this is also the case for some proteomics data, two popular experimental designs in proteomics-sample multiplexing and fractionation-follow different patterns (Fig. 2 ). In multiplexed quantitative experiments (e. g. , based on tandem mass tag (TMT) labeling), multiple samples can be related to the same data file (Fig.
Crowdsourcing annotations of public proteomics datasets Crowdsourcing has been successfully applied to address key problems in bioinformatics  23. The European Bioinformatics Community for Mass Spectrometry (EuBIC-MS  24  , https://eubicms. org) set up a collective effort to annotate existing PX datasets using MAGE-TAB-Proteomics  14. Volunteers from multiple institutes joined the task of annotating, discussing, and improving the SDRF format, openly and collaboratively. Each annotator created an issue in GitHub about a project of interest, forked the main project repository (https://github. com/bigbio/proteomicsmetadata-standard), and annotated the corresponding dataset locally in their computers. Then, a pull request was submitted to include the added annotations. Prior to approval, an independent group of reviewers checked that the proposed SDRF conformed to the guidelines.
This collaborative peer-review system allowed the identification of potential issues in the file format, which is now compatible with the main MS experiment types. Additionally, a file validator was developed to automatically perform a semantic and structural validation of created SDRF files (see next section for details). As of July 2021, over 200 public datasets have been annotated, covering a broad spectrum of organisms, enrichment/fractionation strategies, quantification approaches, and data-acquisition methods (Fig. 3a ). For most multiplexed experiments available on PX, the sample-to-label assignment was not specified by the authors since it was not possible to map the channel to the associated sample in a standardized format. The lack of such essential experimental design information precludes the reprocessing and reproduction of quantitative results.
As a consequence, such datasets are underrepresented in the collection of annotated datasets. This highlights the urgent need for systematic and standardized metadata annotations. To support future annotation efforts, Table  1  shows gold-standard annotated datasets from PX that can be utilized as examples when creating an SDRF file for various experimental designs (e. g. , label-free quantification, multiplex TMT, stable isotope labeling by amino acids in cell culture (SILAC), affinity purification-mass spectrometry (AP-MS), data-independent acquisition (DIA), and phosphopeptide enrichment).
The MAGE-TAB-proteomics toolbox To facilitate automatic data analysis and reuse, we developed a set of software libraries that enable the validation and conversion from MAGE-TAB-Proteomics format to parameter files (Fig. 3b ). For transcriptomics data, the main library to validate a MAGE-TAB file is the Bio-MAGETAB Perl package (https:// metacpan. org/release/Bio-MAGETAB). In principle, Bio-MAGETAB can also be used to validate MAGE-TAB-Proteomics files because they are valid MAGE-TAB files. However, Bio-MAGETAB cannot perform extra validations specific to proteomics data, especially the SDRF-Proteomics rules. To enable the validation of the SDRF-Proteomics of MAGE-TAB-Proteomics files, we implemented two libraries in Java and Python (see below). After validation, parameters such as cleavage agents, post-translational modifications, and mass tolerances (precursor and fragment) are translated into MaxQuant and OpenMS parameters in their corresponding configuration files.
The newly developed Python package, called sdrf-pipelines (https://github. com/bigbio/sdrf-pipelines), enables the validation of the structure and also of the semantic rules applied to SDRF. The sdrf-pipelines package can be installed from different package managers like BioConda 25 or BioContainers  26. It validates the files according to the different experiments and data types, as defined in the templates. For example, if the template corresponds to a human dataset, the software validates that the sample metadata comply with the human template (organism, disease, ancestry, etc. ). In addition, sdrf-pipelines allows users to convert SDRF files to configuration/input files of other popular proteomics analysis tools such as MaxQuant 27 , OpenMS  28  , and MSstats  29  , to facilitate the automation of dataset reanalyses (https://github.
com/bigbio/proteomics-metadata-standard/wiki). The jSDRF Java library (https://github. com/bigbio/jsdrf) we developed also enables the validation of SDRF files. For example, the library can validate that the SDRF contains each sample and data-file relationship, labeling information, fraction identifiers, and the sample and data accessions. It also includes a generic data model that can be used by Java applications to validate and handle SDRF-Proteomics files. While the PX resources have not yet developed any tool to create SDRF-Proteomics files, existing tools for transcriptomics MAGE-TAB and ISA-TAB can be used to facilitate the process. At the moment of writing, we recommend OntoMaton 30 (https:// github. com/bigbio/proteomics-metadata-standard/wiki/Annotating-Sample-terms-using-OntoMaton), which allows the annotation of Google spreadsheets using ontology terms coming from the Ontology Lookup Service (OLS)  31.
Submitters can search ontology terms in OLS and add them to the sample-property columns. The Google spreadsheets provide the functionalities to copy ontology terms across samples, and to add or remove columns and samples. New versions of the tool will include the possibility to add protein modifications, online validation, and loading existing templates among others. Submitting annotated proteomics datasets to the PRIDE database Datasets are standardly submitted to PRIDE using the PX submission tool, a desktop application that guides the users through a set of steps to construct the submission and finally performs the transfer of the data files 2 (Fig. 4 ). The information annotated by the submitter is encoded into a submission. px file. As described Table  1  Examples of MAGE-TAB-Proteomics files.
Dataset type Accession code/hyperlink MAGE-TAB above, this PX XML file is highly similar to the MAGE-TAB-Proteomics IDF file. Therefore, the PX XML file is now automatically converted to IDF using PRIDE internal pipelines, once the submission is received and processed. Since May 2021, the PX submission tool accepts SDRF files. The SDRF file can be created externally as described in the previous section and submitted with the dataset. It is then recognized as an 'EXPERIMENT DESIGN' file type and validated using the jSDRF library (Fig. 4 ). The PRIDE internal pipelines create MAGE-TAB-Proteomics files containing the automatically generated IDF and the user-provided SDRF. Then, the metadata assigned to each sample is automatically submitted to the Bio-Samples database  32. A BioSample accession number is created for each sample in the experiment to enable the linking between samples included in multi-omics datasets.