dataesr/scientific-paragraphs-categorization huggingface dataset:
Study subjects We retrospectively evaluated the model on the overall VENTILA dataset (n = 12,755) as well as a sub-group of VENTILA patients that were admitted with respiratory disorders, specifically COPD, Asthma, interstitial lung disorders, ARDS or Pneumonia (n = 2674). Mortality rate in this subgroup was 36% (n = 960), while the overall morality was 31% (n = 3935).
Statistical analysis and variable selection We use mean and standard deviation to express continuous variables, while categorical variables are expressed as a percentage. No strong linear correlations were found between input variables and the target outcome, as shown in Fig. 1. We carry out four different analysis. Initially we selected six mechanical ventilation parameters as input to our model, namely PaO2_FiO2, peak Pressure, plateau Pressure, applied PEEP, driving pressure and tidal Volume/PBW, as well as age and BMI. We derive the model based on these variables for both: (1) patients in the overall dataset and (2) a subgroup of patients admitted with respiratory disorders. Furthermore, we carry out a sensitivity analyses to evaluate whether inclusion of variables related to the function of other organs, such as such as kidneys (creatinine) and liver (bilirubin) improves the predictive performance of the model.
Dataset pre-processing and missing values handling Datasets were prepared for the analysis in several steps. Initially, outliers and noisy measurements were removed from the data by defining clinically valid intervals for each variable and considering out of interval values as missing values. Secondly, the Fill-Forward imputation methods were applied on each ICU-stay by forward propagating available values to use the nearest valid measurements. Since one of the most common reasons for missingness in ICU data is different frequency of measurements, using the nearest measured value becomes a suitable imputation strategy. Furthermore, imputation of variables completely missing during each ICU-stay was done using median of the variables in the training set. Finally, the data was normalized and scaled to have zero mean and unit variance such that variables with different scales can contribute equally to the analysis.
Machine learning model Our model uses ventilation parameters as an input to predict the likelihood of the patient dying. We restricted the analysis to include ventilation parameters only in order to evaluate their predictive power, while the rest of the variables from this dataset were excluded from the analysis. Patients were included if they had at the variables documented at least once, while the remaining patients were excluded. In terms of machine learning algorithms we chose to evaluate Logistic Regression (LR) as the baseline model, Random Forest (RF), an ensemble of decision trees that has shown great performance in predicting clinical outcomes  [19, 20]  and the Long Short-Term Memory (LSTM) neural network  [3] , a type of Recurrent Neural Network (RNN). As RF and LR are unable to process sequences directly, we expanded each sequence into a single vector that was then fed to RF and LR.
0005. We use Dropout  [21]  to force the neural network to learn a more robust internal representation such that our model can generalise outcome predictions on data of future patients, while we use L1 regularisation method to reduce model complexity and susceptibility to overfitting, increasing generalisability. The performance of each model was evaluated using stratified five-fold cross-validation with 10 times repetition. First, we split the data randomly into model derivation set (80%) and model validation set (20%). Then we built the model and tuned the hyper-parameters based on the validation set. We repeat five-fold cross-validation 10 times on the tuned model to reduce possible bias and evaluate generalisability, where for each run we calculate mean and standard deviation. The LSTM model is implemented using PyTorch  [22]  open source machine learning framework and we also used the scikit-learn software library for the non-RNN models implementation.
Results The overall dataset contained 12,755 ICU stays with complete data on ventilation parameters included in this study, where 3935 ICU stays were recorded as dead (30. 85%). The respiratory disorders subgroup contained 2674 ICU stays with 960 ICU stays recorded as dead. The most common diagnosis for this subgroup was pneumonia (n = 1368) followed by COPD (n = 527), ARDS (n = 501), CPD_nonCOPD (n = 180) and asthma (n = 98). Survivors are compared to non-survivors for the overall patient dataset (Table  1 (a)) as well as the subgroup of patients with respiratory disorders (Table  1.
(b)). We compared our LSTM-based model with both random forrest and logistic regression methods. Predictive performance of LSTM-based model was higher with AUC of 0. 72 and Average Precision (AP) of 0. 57 in comparison to RF and LR for the overall patient dataset. Higher predictive performance of AUC of 0. 75 and AP of 0. 65 was recorded in the subgroup of patients admitted with respiratory disorders, as shown in Fig. 2a, b. Other performance measures, such as PPV, NPV and MCC are detailed in Table  2 (a) and (b), where LSTMbased model outperforms both RF and LR in the majority of performance metrics.
Sensitivity analysis We also performed a sensitivity analysis to evaluate whether inclusion of function of other organs, such as kidney and liver could improve predictive performance of the model. This analysis was carried out for the overall dataset as well as for the subgroup of patients admitted with respiratory disorders, where we included creatinine and bilirubin variables as well as pH and PaCO2, in addition to the variables used for the main analysis. These variables were chosen based on the review of literature as well as suggestion from the clinicians. As it can be seen from Fig. 3a , b, the inclusion of variables related to kidney and liver function increased predictive performance significantly with AUC of 0. 79 and AP 0. 68 for the overall patient dataset and AUC of 0. 79 and AP 0.
Predictive model calibration While the ability of the model to discriminate between patients at higher risk of having an event from those at lower risk is an important aspect, alone it is not sufficient. As such, we also consider model calibration, which measures how similar is the predicted absolute risk to the true observed risk in groups of patients. Poorly calibrated models will underestimate or overestimate the outcome of interest. As such we assess our model calibration by comparing predicted and observed risk of mortality at the whole patient population (mean calibration) as well as a subgroup of patients with respiratory disorders. As it can be seen in Fig. 5 , all models achieved a very good calibration in predicting mortality risk for both the overall patient dataset and the subgroup of patients with respiratory disorders, even though models based on neural networks are typically poorly calibrated, as reported in the literature  [26].
Of course, the cat bites its tail here, as ML benefits especially from large databases with significant patient numbers to exclude noisy data. Albeit the motto "the more, the better" certainly applies here, Shillan et al. were able to show that even for study sizes with 1000-10,000 individuals, satisfactory forecasts with AUCs around 0. 83 can be achieved  [30]. As demonstrated in our study, the use of additional variables can improve test performance and enable high prognostic accuracy in comparatively medium-sized patient collectives. An association between the invasiveness of mechanical ventilation and/or oxygenation indices (especially the P/F value or Horowitz index) and mortality has been shown repeatedly in the past  [14]. Different ventilatory parameters were found to be associated with mortality in previous studies.
Although affected by ventilation strategies and certainly also kidney function, blood pH as a marker of metabolic integrity and partial pressure of carbon dioxide (paCO2) as an indicator of satisfactory ventilation were also included in our model and yielded an even higher predictive performance. It should be mentioned that several challenges could have affected the performance of the model, including the heterogeneity of data, given the multi-centre nature of the dataset spanning across diverse ICUs and countries; consequently, the heterogeneity of data collection protocols; and the averaging of ventilation parameters into a single daily value. As already stated in the past, ML algorithms often lack transparency compared with conventional statistical analyses as they are not as reproducible for most external readers  [30].
Conclusion The result of our analysis has shown that the RNN-based model demonstrated better performance than RF and LR in patients in mechanical ventilation and its subgroup admitted with respiratory disorders. However, it is necessary to validate our results in further studies. We speculate that a dataset with higher granularity-for example, more closely timed records-could lead to an even higher predictive power of AI. The next step would then be to develop algorithms that not only seek to predict outcome, but also suggest alternative ventilation parameters based on prior data, and, for example, seek to ensure even better use and application of evidence-based treatment strategies such as low driving pressure ventilation. If, in a next step, these suggestions-for example, in randomized trials-are associated with a survival benefit for our patients, then a further step would be the development of "closed loop" ventilation systems that seek to optimize the ventilation of critically ill patients on the basis of collected parameters and within evidence-based limits.
Microstructure Analysis of the Stent Wires The local microstructure of stents was examined following the ISO standard for evaluation of cardiovascular implants (International Organization for Standardization, 2016), through use of MATLAB R2019b (Mathworks, Natick, MA, United States). For each treatment scenario, we first obtained a series of images by projecting the stent wires onto the tangent planes at each "node" of the outer-layer stent (Figure  1B ). From each projection, a region of interest (ROI) with four pores immediately surrounding the "node" was then extracted (Figure  1C ). Based upon the ROI, the local porosity (ϕ), pore size (S p ), and pore density (D p ) were calculated as: ϕ = 100 × n i=1 A i A ROI (1) S p = n i=1 A i 2 n i=1 A i (2) where A ROI denotes the area of the ROI; A i refers to the area of the i th pore; and n the number of pores within the ROI.
The LiverTox database compiles cases of idiosyncratic drug-induced liver injury (iDILI) with the promised aims to help identify hepatotoxicants and provide evidence-based information on iDILI. Weaknesses of this approach include case selection merely based on published case number and not on a strong causality assessment method such as the Roussel Uclaf Causality Assessment Method (RUCAM). The aim of this analysis was to find out whether the promised aims have been achieved by comparison of current iDILI case data with those promised in 2012 in LiverTox. First, the LiverTox criteria of likelihood categories applied to iDILI cases were analyzed regarding robustness. Second, the quality was analyzed in LiverTox cases caused by 46 selected drugs implicated in iDILI. LiverTox included iDILI cases of insufficient quality because most promised details were not fulfilled: (1) Standard liver injury definition; (2) incomplete narratives or inaccurate for alternative causes; and (3) not a single case was assessed for causality with RUCAM, as promised.
Instead, causality was arbitrarily judged on the iDILI case number presented in published reports with the same drug. All of these issues characterize the paradox of LiverTox, requiring changes in the method to improve data quality and database reliability. In conclusion, establishing LiverTox is recognized as a valuable effort, but the paradox due to weaknesses between promised data quality and actual data must be settled by substantial improvements, including, for instance, clear definition and identification of iDILI cases after evaluation with RUCAM to establish a robust causality grading.
Introduction The U. S. LiverTox database and website containing preferentially idiosyncratic druginduced liver injury (iDILI) cases became available online in April 2012 and was published in March 2013  [1]. This new approach was much appreciated due to the promising intention to provide not only accurate but also complete summary information on the characteristics of clinical liver injury for each drug, along with an exhaustive and annotated reference compilation. Expectations were high because the website was created by the U. S. National Library of Medicine (NLM) and annotated by the U. S. National Institute of Diabetes and Digestive and Kidney Diseases (NIDDK) staff  [1]. The initial website version was then replaced by a new version  [2]. The data of iDILI cases were said to be collected from various sources, including clinicians submitting case reports  [1].
The website of LiverTox consecutively produced a computer-generated history, associated with a table of laboratory data, and a graphic display of clinical details, which included calculations of latency, time to recovery, severity, and causal relationship by applying the scores of RUCAM (Roussel Uclaf Causality Assessment Method), with various steps, thus ensuring data completeness and high-quality data  [1]. Additional details were provided and seemed overall promising. As a result, expectations among iDILI experts were high for using the database cases as reference cases for clinical and study purposes. However, expectations were only partially met due to problems such as those related to causality assessment, acknowledged by one of the initiators of the LiverTox database and website  [3].
The critical view was in line with views of other groups criticizing shortcomings of the LiverTox initiative  [3] [4] [5] [6] [7] [8] [9] [10]. It seemed that a gap emerged between the promise to provide valid case data of high quality and the presented case data. Most cases are of insufficient quality, especially in terms of a minimum information and a lack evaluation of a causal relationship by applying a strong method such as the RUCAM, a real paradox with a clinical and research impact requiring additional evaluation. The present report analyzed the data quality of selected iDILI cases included in the LiverTox database. It turns out that in a significant number of cases, the presented data are not of sufficient quality to be used in a clinical or research setting. Suggestions of improvement are made, enforcing the application of a strong causality assessment method (CAM) such as the RUCAM in retained cases, whatever the data source.
Materials and Methods A comparison was made between data currently included in the LiverTox database and on its website  [2]  and the aims or promises provided at LiverTox implementation  [1]. First, the LiverTox criteria of likelihood categories applied to iDILI cases were analyzed regarding causality assessment. Second, the data quality was analyzed in iDILI cases selected from LiverTox, which presents on its website all drugs implicated in iDILI as blocks in alphabetic order from A to Z  [2]. From each alphabetical block, the first listed single drug was selected for analysis, excluding groups of several drugs or nondrug products such as herbs. This led to 23 drugs from Abacavir to Zafirlukast implicated in iDILI cases. In addition, and to be on the safe side, a second list of 23 drugs from each alphabetical drug block was analyzed.
LiverTox Criteria of Likelihood Category In the LiverTox database, the causality grading is described in seven likelihood categories applied to iDILI cases  [3, 4]. In other sections of the database, these likelihood categories are termed likelihood "scores," an inappropriate expression that should no longer be used  [2]. Indeed, scores are usually achieved by adding individual scores attributed to specific items in the frame of an algorithm, which is preferentially adherent to Artificial Intelligence (AI) principles  [11] , conditions that do not apply to LiverTox  [1] [2] [3] [4]. Apart from the scoring issue, another problem is the inaccurate definition of the causality categories (Table  1 ).
In other words, with an increasing number of published reports, the causality grading of iDILI cases moved to higher causality levels  [3, 4]. This approach of causality assessment has never been validated and could lead to mistakes and clinical errors. Under these evaluating conditions, among 671 drugs, only 53% were classified as likely causing iDILI, considering the reports in the literature, while 47% were just based on an expert opinion lacking supportive evidence in the literature by previous reports  [3, 4] , as critically discussed  [6]. It was also outlined that although in LiverTox a thorough literature search had been approached, it was not attempted to assess the quality of the published reports or to evaluate the causality of liver injury  [3]. Other analyses of the iDILI cases of the database found that approximately one half of drugs reviewed met the criteria for causing, or being suspected as causing, iDILI  [6, 9].
No efforts had been made to improve case data quality  [18]. Overall, it seems that quality aspects remain a crucial issue of LiverTox. Critical situations emerged when studies used iDILI cases retrieved from the LiverTox database, which may call the study results into question. For instance, an association between iDILI and daily dose, liver metabolism, and lipophilicity has been suggested, but proposals were based not on own valid iDILI cases rather than on cases uncritically retrieved from a variety of drug and DILI databases, including LiverTox  [19] [20] [21]. Such an approach was critically discussed  [22]  in support of the statement that the proposed drug characteristics are not able to predict iDILI with high confidence, associated by a caveat note  [23]. It is obvious that this controversy around risk factors of iDILI is fairly limited to FDA scientists  [19] [20] [21] [22] [23] , an interesting constellation calling for internal solutions  [8, 10, 22].
Long before LiverTox was presented to the scientific community  [1] , national iDILI registries from Spain  [24]  and Sweden  [25]  successfully used RUCAM  [15, 16]  for their cases, viewed as pioneering work and early trust in RUCAM  [7]. As a sign of appreciation, experts discussed only RUCAM-based iDILI and HILI (herb-induced liver injury) cases, rather than cases of the LiverTox database, which were not assessed with RUCAM  [9]. It appears that LiverTox is still far behind mainstream approaches, causing concern about data quality  [1, 2, 8, 10, 18].
LiverTox Paradox Based on Gaps There was much hope among members of the iDILI community when LiverTox entered the challenging field of including valid iDILI cases in a new database and on a website  [1, 2]. Soon, however, it became evident that the applied CAM based on an expert opinion process and the quality of the cases did not meet the expectations (Tables  1 2 3 )  [2] [3] [4] [5] [6] [7] [8] [9] [10]. As a result, the gaps created a paradox: The promise of providing valid case data versus presenting cases of insufficient quality. The identified gaps relate to the completeness and accuracy of the case details, evidence-based features, and causality assessment (Table  4 ).
Promised Data Presented Data and Gaps Cases of iDILI with RUCAM scores  [1]. Evidence is missing that RUCAM was ever used in any iDILI case included in the LiverTox database or presented on the website  [2]. A complete and accurate summary of information about the clinical features of liver injury for each drug  [1]. Clinical summaries were incomplete due to a lack of a diagnostic algorithm such as RUCAM to assess causality  [2]. Instead, causality gradings were arbitrarily published considering the number of published case reports. A website with comprehensive and evidence-based detailed information on iDILI cases  [1]. Information was incomplete and not evidence-based, because the causality was not assessed with a robust method such as RUCAM  [2]  that would have assessed the exclusion of alternative causes.
Suggestions for Improvement Since the time of LiverTox implementation, a variety of suggestions have been made to ensure some degree of high quality of the iDILI cases to be included  [1]. However, the diagnostic causality approach and data presentation of LiverTox remain outside of mainstream opinion (Tables  1  and 2 ). These approaches are not acceptable because they are subjective, not transparent, not structured, not based on strict working procedures leading to variable results, not excluding alternative causes, not validated, and not based on an element scoring allowing for a final causality grading. The results, as presented by LiverTox, are disappointing (Tables  1 2 3 4 ) and have become a matter of debate  [2] [3] [4] [5] [6] [7] [8] [9] [10]. A new approach is now required to improve the quality of the LiverTox database.
Proposals 1. Clinicians, as potential authors of RUCAM-based iDILI case reports, should be encouraged to submit their case reports directly to LiverTox. 2. Additional RUCAM-based iDILI cases should be retrieved from existing iDILI registries of various countries or regions  [10] , including, for example, Sweden  [25] , Iceland  [30] , Spain  [24] , and Latin America  [31] , all of which collect iDILI cases using a prospective approach. 3. Other RUCAM-based iDILI cases should be selected from the 81,856 cases published from 1993 to mid-2020  [26]. 4. From now on, the prerequisites for iDILI cases to be included in the LiverTox database and website should be: • Liver injury must be defined as ALT higher than 5 × ULN (upper limit of normal) and/or ALP higher than 2 × ULN  [12, 27] ; • An informative case narrative with complete diagnostic and clinical details  [27] ; • The R ratio must be calculated based on ALT and ALP values in order to classify the case as hepatocellular injury or cholestatic/mixed liver injury  [27] ; • The case should be assessed with the updated RUCAM  [27] , and the final score should be provided.
Discussion Around one dozen iDILI databases are publicly available with features and limitations well described, whereby the LiverTox database ranks among the major ones and needs to be further improved  [5]. In this context, it has been argued that the interface of LiverTox does not allow for an intuitive approach, impairing for users access to data, and the search request form was found to be limited. Despite these minor shortcomings, there have been many more discussions about the data quality of cases included in the LiverTox database  [4] [5] [6] [7] [8] [9] [10]. Currently, there are criteria problems of causality attribution by not using a robust CAM such as RUCAM, as promised (Table  1 ), and insufficient quality of iDILI cases (Table  2 ), not allowing for appropriate use of included cases by physicians.
Difficult to reconcile is the gap between the initially promised excellence of case data quality and the finally presented quality (Table  4 ). Despite some shortcomings, case quality is much better in national DILI registries using the prospective RUCAM, with a few exemptions  [10]. This suggests their inclusion in the LiverTox database. This analysis may have some limitations; for example, only 52 drugs were randomly included. As with any well-working method in medicine, some background noise is expected for various reasons, especially from scientists, who have never established a robust algorithm such as RUCAM. In this context, several unsuccessful attempts by others to add, modify, or delete elements or to upgrade or downgrade scores were frustrating, reducing the user-friendly handling of the method, making the method more complex and not validated.