dataesr/scientific-paragraphs-categorization huggingface dataset:
Evaluating predictive performance of models For each model that we could validate, we applied the model equation to each individual i in each (imputed) dataset. For each prediction model, we summarised the overall distribution of the linear predictor values for each dataset using the median, interquartile range, and full range, averaging statistics across imputations where necessary  [20]. We examined the predictive performance of each model separately, using measures of discrimination and calibration, firstly in the IPD for each available dataset and then at the meta-analysis level. We assessed model discrimination using the C-statistic with a value of 1 indicating perfect discrimination and 0. 5 indicating no discrimination beyond chance  [21]. Good values of the Cstatistic are hard to define, but we generally considered C-statistic values of 0.
6 to 0. 75 as moderate discrimination  [22]. Calibration was assessed using the calibration slope (ideal value = 1, slope < 1 indicates overfitting, where predictions are too extreme) and the calibrationin-the-large (ideal value = 0). For each dataset containing over 100 outcome events, we also produced calibration plots to visually compare observed and predicted probabilities when there were enough events to categorise participants into 10 risk groups. These plots also included a lowess smoothed calibration curve over all individuals. Where data had been imputed in a particular IPD dataset, the predictive performance measures were calculated in each of the imputed datasets, and then Rubin's rules were applied to combine statistics (and corresponding standard errors) across imputations  [20, 23, 24].
We also compared the performance of the models in the same validation cohort where possible. We used forest plots to show a model's performance in multiple datasets and to compare the average performance (across datasets) of multiple models. A particular challenge is to predict pre-eclampsia in nulliparous women as they have no history from prior pregnancies (which are strong predictors); therefore, we also conducted a subgroup analysis in which we assessed the performance of the models in only nulliparous women from each study.
Decision curve analysis For each pre-eclampsia outcome (any, early, or late onset), we compared prediction models using decision curve analysis  [30, 31]. Decision curves show the net benefit (i. e. benefit versus harm) over a range of clinically relevant threshold probabilities. The model with the greatest net benefit for a particular threshold is considered to have the most clinical value. For this investigation, we chose the IPD that was most frequently used in the external validation of the prediction models and which allowed multiple models to be compared in the same IPD (thus enabling a direct, within-dataset comparison of the models). All statistical analyses were performed using Stata MP Version 15. TRIPOD guidelines were followed for transparent reporting of risk prediction model validation studies  [32, 33].
Results Of the 131 models published on prediction of preeclampsia, only 67 reported the full model equation needed for validation (67/131, 51%) (Supplementary Table  S3 , Additional file 1). Twenty-four of these 67 models (24/67, 36%) met the inclusion criteria for external validation in the IPD datasets (Table  1 )  [35, [46] [47] [48] [49] [50] [51] [52] [53] [54] [55] [56] , and the remaining models (43/67, 64%) did not meet the criteria due to the required predictor information not being available in the IPD datasets (Fig. 1 ).
Characteristics and quality of the validation cohorts IPD from 11 cohorts contained within the IPPIC network contained relevant predictors and outcomes that could be used to validate at least one of the 24 prediction models. Four of the 11 validation cohorts were prospective observational studies (Allen 2017, POP, SCOPE, and Velauthar 2012)  [36, 37, 45] , four were nested within randomised trials  (Chappell 1999 , EMPOWAR,  Poston 2006 , and UPBEAT)  [39] [40] [41] [42] , and three were from prospective registry datasets (ALSPAC, AMND, and St George's)  [38, 43, 44, 57]. Six cohorts included pregnant women with high and low risk of preeclampsia  [37, 38, [43] [44] [45] , four included high-risk women only  [39] [40] [41] [42] , and one included low-risk women only  [36]. Two of the 11 cohorts (SCOPE, POP) included only nulliparous women with singleton pregnancies who were at low risk  [36]  and at any risk of pre-eclampsia  [45].
External validation and meta-analysis of predictive performance We validated the predictive performance of each of the 24 included models in at least one and up to eight validation cohorts. The distributions of the linear predictor and the predicted probability are shown for each model and validation cohort in Supplementary Table  S8  (Additional file 1). Performance of models is given for each cohort separately (including smaller datasets) in Supplementary Table  S9  (Additional file 1).
68, 0. 66, respectively) with minimal heterogeneity when validated in up to six datasets. The summary calibration was suboptimal with either under-or overfitting. When validated in individual cohorts  (Poston 2006 , St George's, and AMND cohorts), the Kuc model showed moderate discrimination in the St George's and AMND cohorts of unselected pregnant women with values ranging from 0. 64 to 0. 68, respectively. But the model was overfitted in both the cohorts (calibration slope 0. 34 and 0. 47) and systematically overpredicted the risks (calibration-in-the-large > 1). In the external cohort of obese pregnant women  (Poston 2006 ), Baschat 2014b model showed moderate discrimination (C-statistic 0. 67, 95% CI 0. 63 to 0. 72). There was some evidence that predictions did not span a wide enough range of probabilities and that the model systematically underpredicted the risks (Table  3 ).
Heterogeneity Where it was possible to estimate it, heterogeneity across studies varied from small (e. g. Plasencia 2007a and Poon 2008 models had I 2 ≤ 3%, τ 2 ≤ 0. 002) to large heterogeneity (e. g. Goetzinger 2010 and Odibo 2011a models had I 2 ≥ 90%, τ 2 ≥ 0. 1) for the C-statistic (on the logit scale), and moderate to large heterogeneity in the calibration slope for about two thirds (8/13, 62%) of all models validated in datasets with around 100 events in total. All models validated in multiple IPD datasets had high levels of heterogeneity in calibration-in-the-large performance. For the majority of models validated in cohorts with a combined event size of around 100 events in total (9/13, 69%), the summary calibration slope was less than or equal to 0. 7 suggesting a general concern of overfitting in the model development (as ideal value is 1, and values < 1 indicate predictions are too extreme).
Summary of findings Of the 131 prediction models developed for predicting the risk of pre-eclampsia, only half published the model equation that is necessary for others to externally validate these models, and of those remaining, only 25 included predictors available to us in the datasets of the validation cohorts. One model could not be validated because of too few events in the validation cohorts. In general, models moderately discriminated between women who did and did not develop any-, early-, or late-onset pre-eclampsia. The performance did not appear to vary noticeably according to the type of predictors (clinical characteristics only; additional biochemical or ultrasound markers) or the trimester. Overall calibration of predicted risks was generally suboptimal. In particular, the summary calibration slope was often much less than 1, suggesting that the developed models were overfitted to their development dataset and thus do not transport well to new populations.
Strengths and limitations To our knowledge, this is the first IPD meta-analysis to externally validate existing prediction models for pre- POP Fig. 3  Decision curves for models of any-onset pre-eclampsia eclampsia. Our comprehensive search identified over 130 published models, illustrating the desire for risk prediction in this field, but also the confusion about which models are reliable. The global IPPIC Network brought together key researchers involved in this field, and their cohorts provided access to the largest IPD on prediction of pregnancy complications. We evaluated whether any of the identified models demonstrated good predictive performance in the UK health system, both on average and within individual cohorts. Access to raw data meant that we could exclude ineligible women, account for timing of predictor measurement and outcome, and increase the sample size for rare outcomes such as early-onset pre-eclampsia.
We could only validate 24 of the 131 published preeclampsia prediction models and were restricted by poor reporting of published models, as well as the unavailability of predictors used in some reported models within our IPD. It is possible that a better performing model exists which we have been unable to validate. However, the issue of missing predictors may also reflect the availability of these predictors in routine clinical practice, and the inconvenience in their measurement, highlighting the need for a practical prediction model with easy to measure and commonly reported variables  [58]. We limited our validation to UK datasets to reduce the heterogeneity arising from outcome definitions and variations in management. Despite this, often considerable heterogeneity remained in predictive performance.
However, the performance of many models continues to be evaluated and compared solely on the basis of their discrimination ability, with calibration ignored  [81]. In the recent ASPRE (Combined Multimarker Screening and Randomized Patient Treatment with Aspirin for Evidence-Based Preeclampsia Prevention) trial  [82] , aspirin significantly reduced the risk of pre-eclampsia in women stratified for high risk of preterm pre-eclampsia using the prediction model by Akolekar 2013  [62]. In the control group, 4. 3% of women were considered to have preterm pre-eclampsia against the 7. 6% expected to be identified by the model. The discrimination of the model was published recently, and its calibration reported in two separate datasets  [83]. The so-called competing risks model appears to have exceptional performance and very high discrimination (> 0.
8) when validated in datasets from a standardised population akin to that used for model development. While this is laudable, caution is needed. The model showed evidence of some problems with calibrationin-the-large and did not examine heterogeneity in calibration performance across centres. Even if all centres across the UK use the same standardisation as the SPREE studies (in terms of timing and methods of predictor measurement), there may still be heterogeneity in the model performance, for example if the baseline risk of pre-eclampsia varied across centres. Therefore, before widespread uptake or implementation of this model, detailed exploration of the performance in a wide range of realistic settings of application is needed, including decision curve analyses. We were not able to validate this model in IPPIC-UK datasets due to lack of information on predictors, and other information needed to calculate the MoMs.
Relevance to clinical practice A clinically useful prediction model should be able to accurately identify women who are at risk of pre-eclampsia in all healthcare settings that the model will be used. There is no evidence from this IPD meta-analysis that, for the subset of published models we could evaluate, any model is applicable for use across all populations within UK healthcare setting. In particular, the poor observed calibration and the large heterogeneity across different datasets suggest that the subset of models are not robust enough for widespread use. It is likely that the predictive performance of the models would be improved by recalibration to particular settings and populations, for which local data are needed. This may not be practical in practice.
This can, for instance, be achieved by comparing the 'local' outcome incidence with the reported incidence from the original development study or by re-estimating the intercept using new patient data. Another important option is to extend the existing models by including new predictors, to both improve the discrimination performance and reduce heterogeneity in baseline risk. To address this, further work could include imputation of systematically missing predictors by borrowing information across studies; techniques for across-dataset imputation are only recently being developed  [90] [91] [92] [93] [94] , and further evidence on their performance is needed before implementation. There is a need to improve homogeneity across studies, for example in predictor measurement method, timing of predictor measurement, and outcome definition.
Conclusion A pre-eclampsia prediction model with good predictive performance would be beneficial to the UK NHS, but the evidence here suggests that, of the 24 models we could validate, their predictive performance is generally moderate, with miscalibration and heterogeneity across UK settings represented by the dataset available. Thus, there is not enough evidence to warrant recommendation for their routine use in clinical practice. Other models exist that we could not validate, which should also be examined in terms of their predictive performance, net benefit, and any heterogeneity across multiple UK settings before consideration for use in practice.
Acknowledgements The following are members of the  IPPIC Collaborative Network + Alex Kwong-University of Bristol; Ary I. Savitri-University Medical Center Utrecht; Kjell Åsmund Salvesen-Norwegian University of Science and Technology; Sohinee Bhattacharya-University of Aberdeen; Cuno S. P. M. Uiterwaal-University Medical Center Utrecht; Annetine C. Staff-University of Oslo; Louise Bjoerkholt Andersen-University of Southern Denmark; Elisa Llurba Olive-Hospital Universitari Vall d'Hebron; Christopher Redman-University of Oxford; George Daskalakis-University of Athens; Maureen Macleod-University of Dundee; Baskaran Thilaganathan-St George's University of London; Javier Arenas Ramírez-University Hospital de Cabueñes; Jacques Massé-Laval University; Asma Khalil-St George's University of London; Francois Audibert-Université de Montréal ;  Per Minor Magnus - Norwegian Institute of Public Health; Anne Karen Jenum-University of Oslo; Ahmet Baschat-Johns Hopkins University School of Medicine; Akihide Ohkuchi-University School of Medicine, Shimotsuke-shi; Fionnuala M.
The spike (S) protein of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) is required for cell entry and is the major focus for vaccine development. We combine cryo electron tomography, subtomogram averaging and molecular dynamics simulations to structurally analyze S in situ. Compared to recombinant S, the viral S is more heavily glycosylated and occurs predominantly in a closed pre-fusion conformation. We show that the stalk domain of S contains three hinges that give the globular domain unexpected orientational freedom. We propose that the hinges allow S to scan the host cell surface, shielded from antibodies by an extensive glycan coat. The structure of native S contributes to our understanding of SARS-CoV-2 infection and the development of safe vaccines. The large scale tomography data set of SARS-CoV-2 used for this study is therefore sufficient to resolve structural features to below 5 Ångstrom, and is publicly available at EMPIAR-10453.
Large scale tomographic analysis of SARS-CoV-2 virions To structurally analyze SARS-CoV-2 spike protein in situ, we passaged the virus through tissue culture cells and purified it from the inactivated supernatant by sucrose centrifugation (see methods for detail). The origin of the SARS-CoV-2 isolate used in this study is south Germany and described in detail in the supplement. We acquired a large scale cryo electron tomography data set of purified virions using acquisition parameters that are suitable for subtomogram averaging (see methods). Visual inspection of the tomographic reconstructions revealed a very high quality data set in which, individual protein domains are clearly visible (Figure  1A , Movie S1). The raw data of 266 tilt-series with >1000 viruses have been deposited EMPIAR-10453 and might be used in the future to explore various structural properties of SARS-CoV-2, in particular its pleiomorphic features.
NovaSTA (https://github. com/turonova/novaSTA) and STOPGAP packages (https://github. com/williamnwan/STOPGAP, see methods for detail). At this resolution, secondary structure elements and individual glycosylation sites are clearly discernible (Figure  2B , C). Although there was no symmetry imposed during the refinement, the average was highly symmetric, suggesting that conformational variability across the three different monomers is minimal. Structural analysis of the asymmetric unit yielded an average map from 65780 subtomograms at an overall 4. 9 Ångstrom resolution. In particular the cluster of parallel helices in the center of the globular domain was nicely resolved (Figure  2D ) and flexible fitting revealed that larger side chains are clearly discernible (Figure  S3 ). Simulated annealing-based classification as implemented in STOPGAP of both, the S-trimer but also the asymmetric unit did not yield any distinct classes.
An unusual coiled-coil resides next to a hinge in the stalk domain of spike While the globular domain was fully contained in the tomographic map, only the directly adjacent part of the stalk domain was resolved, emphasizing the flexibility of the latter. Remarkably, a right-handed coiled coil emerges from the neck of the spike head (~P1140-L1159), forming the upper part of the stalk domain (subsequently referred to as the "upper leg" of the stalk). Right-handed trimeric coiled coils were long thought to be absent from the structural proteome  (Harbury et al. , 1998) , but can be seen in the post-fusion structure of the related mouse hepatitis virus (MHV) spike (PDB ID 6B30;  Walls et al. , 2017a). The density in the single-particle structures of SARS-CoV-2 spike  (Wrapp et al. , 2020) , albeit only moderately resolved in this region next to the trimerization tag, is consistent with a righthanded coiled-coil.
The assignment as right-handed coiled-coil is also consistent with the observation that this sequence stretch is not part of the post-fusion left-handed coiled-coil, as resolved for SARS-CoV spike (PDB ID 1WYY;  Duquerroy et al. , 2005) , which indicates a certain incompatibility with left-handedness. The observed electron density sharply declined at a defined position within the trimeric coiled-coil, suggesting at least one flexible hinge (Figure  2B ). We analyzed the conformational variability of the stalk domains based on the position of the globular domain and its orientation with respect to the membrane as defined by subtomogram averaging. The globular domain exhibited large positional and orientational freedom, tilting up to ~90 degrees with respect to the normal at distances of 5-35 nm from the membrane without discernible correlation (Figure  2E ).
A particularly unusual feature masked at the edge of the resolved density of the single-particle structures but well resolved in the subtomogram averages is the short right-handed coiled-coil at the top of the pre-fusion stalk. Being lost in the post-fusion structure as resolved for SARS-CoV (PDB ID 1WYY;  Duquerroy et al. 2005) , we speculate that the right-handed coiled-coil is only marginally stable, priming the protein for a large structural reorganization in a spring-loaded viral fusion mechanism. These findings underline that all three hinges become irrelevant during the transition to the post-fusion conformation, which places them outside the structural core  (Cai et al. , bioRxiv 2020; Duquerroy et al. , 2005). A remarkable difference between SARS-CoV and SARS-CoV-2 spike is the presence of a furin cleavage site in the latter  (Wrapp et al.
Acknowledgments The cryo electron tomography data was collected at the  EMBL Heidelberg Cryo Electron Microscopy Service Platform. We thank  EMBL (BT, WH, SM, AS, MB) , the  Max Planck Society  ( BT, MS, SW, FECB, SvB, MG, SM, RC, GH, MB ), the  Max Planck Computing  and  Data Facility  for providing computational resources. We also acknowledge a generous SuperMUC-NG computing allocation at the  Leibniz Supercomputing Centre. (MS,SvB,MG,FECB,RC,GH), the  Human Frontier Science Program  ( RGP0026/2017 ; SvB, GH), the  German Ministry of Health (CS) , the  German Center for Infection Research (CH,MDM)  and the Loewe centre DRUID from the  Justus Liebig university Giessen (JKL)  for funding. BT acknowledges  William Wan  ( Vanderbilt University ) for helpful discussions. JKL acknowledges excellent support by  Regina Eberle  (PEI).
Data availability All reagents are available upon request. B6. Y POS sequence data analyzed here have been submitted to the NCBI (BioSample accession: SAMN12896768). Supplemental material available at figshare: https://doi. org/10. 25386/genetics. 11323655. Each blue point represents a detected SNP at a given chromosome position. The y-axis of each chromosome plot shows the base-quality scores (BQS) for each SNP, ranging from 200 to 1000. The largest cluster of heterozygous SNPs was found on chromosome 13 between 38 and 76 Mb. Much smaller regions exhibiting a cluster of non-B6 SNPs could not be independently validated and were attributed to sequence context, sequencing errors or genetic drift. Note also the large number of SNPs on the Y chromosome, revealing differences between the M. m. domesticus Y POS chromosome and the C57BL/6J Y chromosome.
d. poschiavinus sequences in the mouse genome database; or, more likely, at some point after the establishment of the B6. Y POS strain at Harwell, an outcross was performed, to a strain thought likely to be protective against sex reversal, such as BALB/cJ, in an attempt to rescue a, perhaps small, colony in which very few B6. Y POS fertile males were being produced-this might be the fate of any colony lacking Mod13 or another modifier/transgene. F1 males from such a cross would then have been crossed to B6 for subsequent generations, gradually resulting in the B6 background being restored, but with segments of the introgressed Mod13 region being maintained. However, we can find no evidence of such an outcross in the breeding records at Harwell. In this context, it is interesting to note that several of the inbred strains showing sequence similarity to Mod13 variants, such as BALB/cJ, C3H, and DBA, are known to protect against the disruptive effects of Y POS , i.
Acknowledgments We thank the husbandry team in Ward 5 of the  Mary Lyon Centre (MLC) at Harwell , particularly  Jackie Harrison ,  Gemma Atkins , and  Michelle Sandell. We also thank  Martin Fray  and the  MLC (Medical Research Council) FESA (Frozen Embryo & Sperm Archive ) team. We thank  Valerie Arboleda  and  Eric Vilain  for kindly providing tissues and genotyping details for the chromosome 11 modifier. We dedicate this paper to  Bruce Cattanach , whose pioneering research on mouse sex determination at Harwell remains an inspiration.
Data Quality Control The raw data obtained from MS detection were uploaded directly into proteome discovery v. 2. 2 (Thermo Fisher Scientific, Waltham, MA, USA) software for database retrieval, peptide mapping and protein quantification. The retrieved results were filtered using proteome Discoverer v. 2. 2. Peptide Spectrum Matches (PSMs) with 95% confidence intervals. Proteins containing at least one unique peptide fragment were considered reliable. The reliable PSMs and proteins were verified with other reliable proteins. Peptide fragments and proteins with false discovery rates (FDR) of >5% were excluded. The protein sequences of sugarcane infected by X. albilineans were deposited into the United States National Center for Biotechnology Information (NCBI) SRA database under accession number PXD015930.