dataesr/scientific-paragraphs-categorization huggingface dataset:
g. , detection and segmentation of a healthy lung within a chest CT scan) will require less training data than the detection of subtle, small changes in a whole-body CT scan. The less training data is available, the better the quality of the data should be. Large training data sets allow for some inaccuracies. It is important to keep in mind that training data quality is a limiting factor for an algorithm's performance. Therefore, the required data quality also depends on the envisaged performance of the algorithm. The decision how to obtain ground truth again requires both clinical and technical expertise. It revolves around questions such as how data can be extracted from hospital information systems, whether or not to use public databases, and how and by how many experts the ground truth should be established.
Whenever patient data leaves primary clinical systems, it is of upmost importance to ensure complete de-identification. In radiology, this includes erasing or overwriting all DICOM tags that contain data privacy relevant information. It is highly recommended to double-check the success of this de-identification process by reviewing DICOM metadata before sending image data. An approach that allows for collaborative training of an ML model without exchanging data samples is "Federated learning"  [8] : a copy of an algorithm is downloaded and local data used to further improve it. The resulting changes to the model are summarized in an update that is uploaded and merged with the central consensus model. Preferably, ground truth data is stored in interchangeable formats (e. g. , the Neuroimaging Informatics Technology Initiative (NIfTI) data format for segmentations) to assure usability for other projects.
This is also in accordance with the FAIR guiding principles for scientific data management and stewardship  [9]. Unfortunately, so far no standards have been established for ML and most clinically used post-processing software is not capable of exporting segmentation masks, which limits the reusability of data and impedes reproducibility of studies. There are many other types of ground truth labels, comprising labels on the level of a whole dataset (fracture on radiograph: yes/no) and outcome labels for prediction modeling (patient death: yes/no). In cardiovascular radiology, most algorithms solve segmentation tasks; therefore, segmentation masks are the predominant type of ground truth in this field. However, also tissue characterization (e. g. , T1 and T2 mapping, late gadolinium enhancement) is an increasing part of cardiovascular radiology and needs adequate ground truth labels, e.
Deep Learning (DL) Subset of ML algorithms consisting of multiple calculation layers that permit the extraction of high-level features biases introduced by the composition of datasets is important: an algorithm for clinical outcome prediction developed on a training dataset containing 80% males from country A might not work well on data from female patients in country B. In general, the dataset on which an algorithm is developed should reflect the population on which the algorithm is later applied as good as possible. Either one is aware of these limitations or overcomes the challenge by using large, heterogeneous datasets. The gold standard is to evaluate an algorithm's performance in clinical practice and their influence on clinical workflows. Finally, a suitable ML technique has to be selected (e.
• Experts involved in ML development should make use of online resources for creating, sharing, and discussing ML algorithms. Besides human expertise, ML projects have requirements with regard to hardware and software. Hardware Standard CPUs are sufficient to run most non-DL ML algorithms and even DL approaches like deep convolutional neural networks (DCNNs) with few layers. However, DCNNs with multiple layers, which constitute the majority of currently developed ML algorithms of interest for cardiovascular imaging, are more computationally intensive. These algorithms need dedicated hardware with GPUs. Commercially available consumer GPUs with 8 GB or more system memory currently suffice for many applications. A detailed overview on hardware for ML, its performance, and pricing is provided by Tim Dettmers  [12].
Alternatively, data can be processed using off-site cloud solutions such as Amazon Web Services. Software The ML community is fully digital and publishes mostly open source. Practically all relevant resources like software libraries and discussion forums are freely accessible. Jupyter Notebook is a commonly chosen web-based platform to compile ML algorithms (jupyter. org). The platform allows the use of multiple programming languages, including Python, which currently is the most prevalent language in the field of ML (python. org). A programming language can be thought of as the vocabulary and rule system that is used to instruct a computer to perform tasks. ML algorithms can be developed using software libraries like TensorFlow (tensorflow. org), scikit-learn (scikit-learn. org), and PyTorch (pytorch.
org). These libraries contain pre-written code and procedures that enable easier and faster software code development. Other alternatives are MATLAB (mathworks. com) and R (www. r-project. org). Once the code is created, it should preferably be shared publicly. GitHub is a common online Git repository for sharing and discussing software code with version control function that allows to retrace a project's source code history (github. com). Furthermore, anonymization tools are important for ML projects in radiology, because sensitive patient information is part of the DICOM header of each image and data exchange is needed to build large databases with studies from multiple centers.
Sample size motivation In contrast to the recommendations made in the STARD and CONSORT guidelines, most ML studies have not explicitly considered sample size when designing the study and are often based on convenience samples. However, sample size and a statistical analysis plan should ideally be prespecified. Although there are presently no clear guidelines on how to calculate a sample size in ML studies, the number of subjects or datasets can be prespecified according to considerations such as the minimal clinical difference of interest or the expectation that ML is able to generate equivalent results to human observers on a certain task. Furthermore, sample sizes used by other researchers to solve comparable problems might be a good indicator.
Specification of study design and training, validation, and testing datasets Algorithm development demands data for training, validation, and testing. Investigators should specify how the data was split into each of these categories. It is of utmost importance to strictly separate the testing dataset from the other datasets to obtain a realistic estimate of model performance. This is also a requirement for regulatory approval of MLbased computer-assisted detection devices from the United States Food and Drug Administration (FDA)  [19]. Ideally, validation is performed not only on internal data (from the same department or institute) but also on an external dataset by independent researchers.
Can the results be applied in a clinical setting? Machine learning studies designed to solve a specific clinical problem should explicitly consider whether the results apply to a real-world clinical setting. This includes discussion of how representative the dataset used for derivation and testing of the model is of the clinical setting in which it will be applied. Any sources of bias, in particular class imbalance and spectrum bias, should be identified and discussed. Considering these factors can enable more precise identification of patients in which the algorithm can be used clinically, or in which groups of patients and clinical scenarios additional validation is needed. Investigators should also consider if and □ Were existing labels used, or were labels newly created for the study? □ How many observers contributed to the standard of reference? □ Were observers blinded to the output of the ML algorithm and to labels of other observers? 6.
Reporting of results □ Which measures are used to report diagnostic or prognostic accuracy? □ Which other measures are used to express agreement between the ML algorithm and the standard of reference? □ Are contingency tables given? □ Are confidence estimates given? 7. Are the results explainable? □ Is it clear how the ML algorithm came to a specific classification or recommendation? □ Which strategies were used to investigate the algorithm's internal logic? 8. Can the results be applied in a clinical setting? □ Is the dataset representative of the clinical setting in which the model will be applied? □ What are significant sources of bias? □ For which patients can it be used clinically? □ Can the results be implemented at the point of care? 9. Is the performance reproducible and generalizable? □ Has reproducibility been studied? □ Has the ML algorithm been validated externally? □ Which sources of variation have been studied? 10.
Is performance reproducible and generalizable? To date, in most reports on ML, model development, tuning, and testing have been performed on a convenience sample of locally available data. Although many of these reports have demonstrated encouraging results, it is important to investigate the reproducibility of the model and to perform an external validation, preferably on multiple datasets from other independent institutes and investigators. External validation is important to Table  2  Example of applying the checklist to the research article "Automated cardiovascular magnetic resonance imaging analysis with fully convolutional networks" by Bai et al  [17]  1. Which clinical problem is being solved? The study by Bai et al is focused on the fully automated determination of left ventricular ejection fraction (LVEF) and right ventricular ejection fraction (RVEF) on cardiovascular magnetic resonance imaging data, without the need for contour drawing by human experts.
Choice of ML model The investigators used a DL approach with a fully convolutional neural network architecture consisting of 16 layers. Separate networks were trained for three commonly acquired anatomical orientations (short axis as well as vertical and horizontal long axes). Detailed parameters regarding network training are provided. The most important measure to look for with regard to overfitting is the use of strictly separate or "hold out" test dataset (see item 4 below). There was no explicit mention of any other measures taken to avoid overfitting.
Specification of study design and training, validation, and testing datasets A random sample of the British population participating in the UK Biobank study was used. As such, this is a retrospective cross-sectional analysis focused on understanding variations in the LVEF in the general population. Detailed inclusion and exclusion criteria were provided. The number of patients used for training, validation, and testing was 3975, 300, and 600 for the short axis segmentation algorithm; 3823, 300, and 600 for the vertical long axis algorithm; and 3782, 300, and 600 for the horizontal long axis algorithm. The investigators do not explicitly mention whether the test dataset was kept separate from the development and validation datasets. The developed algorithms were tested in 1734 additional UK Biobank participants.
Is the code available? The cardiac MR data including the segmentations are available upon request for health-related research in the public interest. The software code is available on GitHub. It is unclear if the algorithm needs to be retrained with new data. investigate the robustness of the model to e. g. differences in image acquisition and reconstruction methods between different vendors and institutes and differences in referral patterns and variability in the prevalence of the condition of interest. Conversely, we also believe it is advisable to validate external algorithms prior to local use, especially if the algorithms`results are used for automated analysis with results directly transferred into clinical reports instead of use as a second reading tool. 10. Is there any evidence that the model has an effect on patient outcomes? Although one of the first proofs of concept in the development of an ML algorithm is the investigation of its diagnostic accuracy, investigators and readers should ask themselves the question whether there is any evidence of an effect on patient outcomes.
Insights of a systematic literature review on applications of ML in cardiac radiology To identify articles on the application of ML in cardiac radiology, a comprehensive search for articles in PubMed and EMBASE databases was conducted. The search identified all articles in the English language registered no later than 31. 01. 2020 (n = 599 in PubMed; n = 2559 in EMBASE). Supplement 1 documents the search strings.
• Further challenges encompass rare disease and/or anatomical variants and compliance with legal frameworks. The compilation of high-quality datasets for ML projects in radiology is hampered by some peculiarities of health-related data. First, there is the issue of significant inter-and intrareader variability fostered by the fact that many categories in medicine are not as distinct as those of everyday objects such as dogs or cars. Second is the complexity of medical image interpretation. For example, a small hyperintense streak in late gadolinium enhancement imaging may be a hyperintensity artifact or a true scar. For clarification, one needs to integrate additional information such as whether there is an implanted cardiac device or not. Third, there is a lack of standardization in the acquisition of medical data.
Solutions to this challenge are either complex, or reduce information (processing of a 3D CT dataset as a series of multiple 2D images). Another important challenge to ML projects in radiology is strict standards of data privacy. All this makes the creation of high-quality ground truth datasets in healthcare challenging and expensive. As a result, datasets in healthcare-related ML projects tend to be much smaller compared to the non-medical domain: the famous contest for everyday object detection, the ImageNet challenge, encompasses over 14 million images (http://www. image-net. org/about-stats), while ML studies in cardiovascular radiology often comprise less than 100 cases. Public datasets like the ChestX-ray8 dataset provided by the NIH containing more than 100,000 frontal-view radiographs with eight disease labels are important initiatives to overcome this problem.
However, labels need critical quality review, which requires medical domain knowledge. For cardiovascular radiology, comparable datasets are currently lacking and professional societies can play an important role in the assembly of large publicly available datasets with high-quality ground truth labels to allow for an objective comparison of different ML algorithms. Registries like the ESCR MR/CT registry (https://www. mrct-registry. org), providing large standardized data sets for further analyses with currently > 300,000 deidentified examinations  [26] , are also an important contribution in this direction. Apart from image data-related problems, there are other challenges. First, there is the problem of rare disease entities. ML algorithms need a sufficient amount of training examples to detect patterns, ideally including examples at the extreme ends of the disease spectrum.
However, many radiologic disease patterns are rarely seen, like congenital heart diseases, and ensuring a fully representative training dataset remains a difficulty. Second, legal issues: Machine learning algorithms, although highly accurate for many tasks, are never perfect and discussions on legal liability for incorrect or missed diagnoses are ongoing  [27]. Third, the question of acceptance: will physicians and patients be willing to trust judgments of algorithms that are a "black box" to them? However, trust in systems not fully understandable to us is part of day-to-day life.
Firstly, we compared the risk of cumulative incidences of allcause mortality and graft loss with a subdistribution hazard competing risk model (Fine and Gray). Secondly, respective performances of mGFR and eGFR changes as predictors of 16-year post-transplant outcomes were analysed by comparing: (i) In terms of discrimination, C-statistics from time-dependent ROC AUC. (ii) In terms of calibration, subdistribution hazard ratio (HR) of all-cause mortality and graft failure. A P-value of 0. 05 was the level for statistical significance in all analyses. Patients lost to follow-up were censored at latest news. SAS software version 9 (SAS Institute, Cary, NC, USA) and R package 'https://cran. r-project. org/web/packages/timeROC/ timeROC. pdf' were used to calculate and compare C-statistics.
The sankey diagram shows how human orthologues of mouse genes with IMPC primary viability assessment (lethal, subviable and viable) regroup into essential and non-essential human cell categories; the width of the bands is proportional to the number of genes. b Gene Ontology Biological Process (GO BP) enrichment results. Significantly enriched GO terms at the biological process level were computed using the set of IMPC mouse-to-human orthologues incorporated into the FUSIL categories as a reference (Table  1 ) and identified after correcting for multiple comparisons. Significant results were only found for the cellular and developmental lethal gene categories. Bubble size is proportional to the frequency of the term in the database and the colour indicates significance level as obtained in the enrichment analysis.
2c ) or the likelihood of the gene product being part of a protein complex (Fig. 2d ). CL genes also stand out as a singular category regarding the number of paralogues (Fig. 2e ). In contrast, features associated with mutational rate appear to peak in the DL or SV bins: probability of mutation based on gene context (Fig. 2f ), transcript length (Fig. 2g ), and strength of negative selection measured by Gene-level Integrated Metric of negative Selection (GIMS) scores (Fig. 2h ). However, these observed increases in the DL and SV genes relative to CL were only statistically significant for transcript length (Supplementary Table  4 ). A similar effect is observed with several intolerance to variation scores from the Genome Aggregation Database (gnomAD). Higher (but not statistically significant) probability of LoF intolerance (pLI) values were observed in the DL and SV genes relative to CL, with a lower frequency of pLI values close to 0 among the viable bins when compared to the CL, DL and SV sets (Fig.
3d ). The degree of pleiotropy, measured by the number of physiological systems affected according to HPO annotations, followed a similar pattern, with CL and DL genes showing the highest number of affected systems (Fig. 3e ). In summary, our results suggest that disease genes in the DL fraction correlated with earlier age of onset, multiple affected systems and AD disorders (Supplementary Table  6 ). Given this, we compared the genes in our FUSIL bins with the genes identified as likely to be causative of developmental disorders in the DDD Study as reported in the Development Disorder Genotype-Phenotype Database (DDG2P) and found a strong enrichment for these genes in the DL FUSIL bin (3. 9-fold increase; Fig. 3f ). The DDDG2P resource has a larger representation of non-consanguineous patients with de novo mutations compared to the OMIM and Orphanet resources  37  , so we next compared enrichment of DL genes by mode of inheritance across the three resources and found the strongest enrichment for monoallelic disease genes in the DDDG2P resource (Fig.
We next focussed on unsolved diagnostic cases from three large rare disease sequencing programmes to investigate potential disease candidates within our set of 163 prioritised DL genes (Supplementary Data 4). First, DDD makes publicly available a set of functional de novo variants and/or rare homozygous, hemizygous or compound heterozygous LoF research variants of unknown significance found in genes that are not associated with human disease according to OMIM and DDG2P. These variants were found in 4293 children with developmental disorders who participated in the UK DDD study and remained undiagnosed. Given recent findings from DDD that most developmental disorder cases could be explained by de novo coding mutations  37  , we searched for heterozygous, de novo variants in this data set that affected any of the prioritised 163 DL candidate genes described above and found variants in 44 genes that met these criteria.
Second, we searched 18,000 rare disease cases from the 100KGP  31  and discovered de novo variants in undiagnosed patients with intellectual disability in 47 of the 163 genes in our candidate set. Lastly, the CMG, a collaborative network of centres to discover new genes responsible for Mendelian phenotypes provided a list of phenotypes studied and potential associated genes  39  , which included around 2000 genes. A set of 14 genes overlapping with the 163 DL candidates are classified as either Tier 1 or Tier 2 genes (see "Methods"). We additionally explored denovo-db, a database containing de novo variants identified in the literature  40  , and annotated our set of 163 prioritised genes with this information. We found at least one functional de novo variant reported in the database for 108 (66%) genes, of which 83 (51%) contained entries collected from resources other than DDD 2 (Supplementary Data 4).
There was some degree of overlap between the candidates identified in the three programmes (Fig. 4a ) and for the next stage we focussed on genes with evidence from both the 100KGP (where we had detailed patient phenotypes and variant information) and either DDD (variants and high-level phenotypes available) or the CMG (gene and high-level phenotypes available). We particularly focussed on 9 genes where the associated variants were not present in any population in gnomAD and each gene was also not highly tolerant to missense variation (o/e missense < 0. 8; Fig. 4b , Supplementary Data 4). For these genes, further evidence for candidacy was gathered based on the number of unrelated families and phenotypic similarities between them, protein-protein interactions with known developmental disorder genes, embryonic and adult mouse gene expression in relevant tissues and embryonic and adult mouse phenotypes that recapitulate the clinical phenotypes.
Median logTPM expression values from the GTEx database for selected non-correlated tissues are shown. c Protein-protein interaction network parameters. Notched box plots showing the distribution of degree and topological coefficient computed from human protein-protein interaction data extracted from STRING. Only high-confidence interactions, defined as those with a combined score of >0. 7, were kept. d Protein complexes. Bar plots representing the percentage of genes in each FUSIL bin being part of a protein complex (human protein complexes). e Paralogues. The bar plot shows the percentage of genes without a protein-coding paralogue gene in each FUSIL bin. Paralogues of human genes were obtained from Ensembl Genes 95. A cut-off of 30% amino acid similarity was used. f Probability of mutation.
Distribution of gene-specific probabilities of mutation from Samocha et al. 65. g Transcript length. Maximum transcript lengths among all the associated gene transcripts (Ensembl Genes 95, hsapiens data set). h GIMS Selection Score. Distribution of Gene-level Integrated Metric of negative Selection (GIMS)  66  scores across the different FUSIL bins. i Probability of loss-of-function intolerance (pLI) retrieved from gnomAD2. 1. Notched box plots and density plots showing the bimodal distribution of this score, with higher values indicating more intolerance to variation. j Distribution of gnomAD o/e LoF scores. Upper bound fraction of the confidence interval around the observed versus expected LoF score ratio (gnomAD 2. 1. ). A score <0. 35 (dashed line) has been suggested to identify intolerant to LoF variation genes  56.