dataesr/scientific-paragraphs-categorization huggingface dataset:
Additionally, being a generative model, it does not require an explicit discriminatory function and no diagnosis or outcome data was used in the model, or in its training or assessment. One can consider the HMM technique as a dimensionality reduction of the multivariate space of patients, which has a natural time dependence. In this case, transforming MVTS data into a simpler state space which maximises the likelihood of the observed data and captures some dominant aspects of the patient's condition on each day. Thus, inpatients' trajectories are represented as a univariate discrete time series, with each day of each AE represented as a unique state space, instead of 23 numeric variables. An HMM with Gaussian emissions, full covariance matrix and 17 states was trained and fitted in the entire 'training and test' dataset with the Python package hmmlearn.
The number of states set in the model was selected in a 2-fold cross validation process where the 'training and test' set was randomly divided into two subsets, which were separately used to train a model and then fit the entire dataset. The output of both models was compared on a concordance matrix of co-occurrences for patient and day with the aim to find an association between states in both output datasets (i. e. , a model trained with n states in two different datasets would lead to similar results or predicted states), while selecting a number of states large enough to capture the heterogeneity of the data. Chi-square tests showed that the pvalues for all tested numbers of states (2 to 50) were significantly different (p-value <0. 0001). The visualization of Pearson residuals and contribution plots (the contribution of a given cell to the total Chi-square score) showed that 17 states provided good concordance between states pairs without being too restrictive (Supplementary Figure  2 ).
A clinician (VLK) first inspected the distribution (mean values and variance) of each of the vital signs and laboratory test values stratified by state. This information was combined with a visual evaluation of: 1) the temporal distribution of the state within AE; 2) the distribution of the state across AE organised by the primary admission diagnosis; and 3) the distribution of the state in those discharged alive compared to those who died during the inpatient episode. Then formal association analyses were conducted using Chi-square tests to confirm the expert interpretation. Reproducibility of the results was assessed by re-training the HMM algorithm in the 'training and test' dataset and fitting the model to the 'hold-out validation' dataset. The distribution of the variables was compared to map the resulting states to those in the initial training and Pearson correlation between their means was calculated.
Prediction models in the first days of admission In addition to the generative modelling, we also wanted to explore the best timeframe and dataset for making discriminatory decisions. This is an obvious use of EHR data and clinically it is of interest to identify patients at higher risk early after admission. Two well established fixed frame ML techniques were employed, logistic regression (LR) and random forest (RF). LR and RF models were trained to evaluate their performance on the prediction of AE outcome. Specifically, the impact of combining patients' representations during the first three days of admission (either the MVTS data or the HMM state spaces) with information on age, sex, CFS score and primary admission diagnosis was assessed and compared. LR models were trained for binary classification only (i.
e. , prediction of IM) and RF models for binary, multiclass (prediction of five disjoint outcome classes: ID, DA, PDR, PDRM and PDM; and prediction of primary diagnosis at admission [PDA]) and multilabel classification (prediction of diagnoses at discharge [DD]). The 'training and test' dataset was divided into a training set (80%) for 5-fold cross validation hyperparameter tuning and a test set (20%) to evaluate the performance of the model with the selected best parameters. The model was trained in the training set and evaluated in the test set ten times to calculate the mean of the performance metrics. All the trained models were then run against the 'hold-out validation' dataset without additional training.
RESULTS: 3. 1. Patient cohort: characteristics and evaluation of data processing The initial dataset included more than 146 000 AE from 61 513 patients. After inclusion criteria, the final cohort comprised 11 158 unique AE equating to the same number of unique patients. The cohort characteristics are summarised in Supplementary Table  2. Data was imputed for 25. 5% of laboratory test values (7. 8% MI; 17. 7% LI) and 0. 3% (0. 26% MI; 0. 04% LI) of vital sign values in the 'training and test' dataset. Kernel density estimates of the imputed and observed data for each variable are shown in Supplementary Figure  3A. The distributions match well for all variables except those with very small numbers of missing observations (i. e. , 4 to 36 imputed values). MI is an iterative process and convergence plots show the mean and standard deviation of the synthetic values plotted against iteration number for the imputed data (Supplementary Figure  3B ).
This is similar to the clinical description of the Systemic Inflammatory Response Syndrome (SIRS)  [29]. In general, these states are present in several diagnostic codes but not strongly over-represented in any one ICD-10 code (Supplementary Figure  5 ). Not all 17 states could be classified confidently. For example, some states such as 'Stable but Static Renal-like' state and 'Other Illness Presentation-like' state did not have a clear 'predominant feature' (Supplementary Table  3 ). AEs trajectories as HMM-LTVS states (17 states)  The algorithm was trained again in the 'training and test' dataset and the resulting model used to assess the 'hold-out validation' dataset. The distributions of the laboratory test result and vital signs in both outputs showed similar patterns and allowed the mapping of all states with the exception of 'Other Illness Presentation-like' state, for which some variables did not match as well as the for the other states (Supplementary Figure  6 ).
The percentages of patients classified in this way were 80% and 86% in the 'training and test' and 'hold-out validation' datasets, respectively. Formal statistical tests were used to evaluate associations of each state with sex, age-group, CFS category, month of admission, admission diagnosis, discharge specialty, length of stay, number of diagnoses at discharge, and hospital outcomes (IM, PDR, PDM), none of which were variables used to train the HMM. Some trends could be predicted by the initial visual clinical interpretation of the states and supported the expert interpretation. For example, 'Disease-like' states showed the strongest associations with admission ICD-10 codes and overall, 12 States (71%) showed at least one association with an ICD-10 code (p-value <0. 001 with Bonferroni correction) and a similar pattern of associations was also observed in the 'hold-out validation' dataset (Figure  9 ).
performance) are summarized in Supplementary Table  5. The trained models were assessed with the 'hold-out validation' dataset. For simplicity, in this paper we describe only the performance achieved by the best models as identified in the 'training and test' dataset. Detailed results of performance metrics in both datasets are reported in Supplementary Tables  (6) (7) (8) (9) (10) (11) (12) (13) (14) (15). Although similar, higher performance on ROC-AUC and weighted ROC-AUC was observed overall for models trained with MVTS compared to states representations (Table  2 ). Also, similar performances were observed by group, especially in the models trained with states data. Almost all models showed highest performance for all metrics when trained with either Day 3 (D3) or Days 1 to 3 (D1D2D3) input variables and the inclusion of phenotypic information modestly improved the performance of models trained to predict inpatient mortality and clinical outcome at 30-day of admission.
Prediction of inpatient mortality was assessed comparing the performance of RF and LR models (Supplementary Tables  6 7 8 ). While only the RF trained with MVTS data (with or without phenotypic information) achieved reasonable precision compared to other models (0. 72 and 0. 62, respectively), recall results were very low (highest 0. 2). The opposite was observed with the RF models trained with HMM states (highest precision and recall 0. 2 and 0. 7, respectively). Recall obtained by LR models was higher than by RF models, which explains that the highest F1-score was achieved with a LR model with MVTS data (Supplementary Table  8 ). Consistent results were observed in the hold-out validation data. However, the low recall of the best model in the test dataset (only 14 patients predicted at higher risk of inpatient death in five or more runs out of 226 real cases) limits the analysis of these results.
Feature ranking for the best RF model (based on ROC-AUC performance) showed that the top five features were respiratory rate, eosinophil count, urea, lymphocytes and neutrophils. These features closely resemble those characterising the 'Early Inflammatory Response-like' state (high WBC, neutrophils, urea, respiratory rate, and heart rate; Figure  7 ), which emerged from the HMM and was associated with inpatient mortality in both the visual clinical interpretation and statistical analysis of HMM state associations (Supplementary Figure  7C ). We have provided one approach addressing some of these issues and found the most important aspects of data preprocessing were regularisation and imputation of missing values. Daily bins across time series records of >3 days were selected because of our interest in blood biochemistry as input variables, which were recorded once per day for most tests and patients.
Regularisation of time series is a necessary step for many ML models but implies loss of information for those tests with multiple recorded observations on a pre-established bin and the introduction of missing values where there are no recorded observations. In future, we will explore different ways of feeding or representing this information to analyse its impact on inpatient representations or outcome prediction  [30]. Our approach towards imputation was a two-step process. MI was the preferred method because it takes advantage of relationships between variables in the dataset  [31]  but this method cannot be used on days with many missing variables. In that case, LI was used at the individual time series level. To reduce the impact of imputed values on our results, strict inclusion criteria were used to select the final cohort of patients, aimed to limit missingness and avoid imputation of large numbers of consecutive missing observations.
Furthermore, significant numbers of states were associated either with specific diagnosis codes or stages of the AE. These findings are all made with no explicit training data for outcomes or diagnoses, but rather emerge from the generative model of the HMM and strongly suggest the HMM is capturing real patient biology. We acknowledge that the HMM model which insists on a single state being assigned to a patient day here is a crude approximation to clinical reality, but nevertheless reassured that even with this constraint the resulting HMM was a useful model to explore clinical trajectories. Little emphasis has been placed on simply exploring data extracted from hospital EHR systems. Instead, this data is used to develop prediction models for pre-specified outcomes  [8]. Whilst this focus is understandable, given its potential clinical utility, information is lost pertaining to the variability of what happens to patients after emergency hospitalisation.
This code includes 'injuries' and will capture older patients hospitalised due to the consequences of a fall e. g. , hip fracture. Falls and associated injuries are key consequences of frailty, place significant burden on the NHS  [41]  and are associated with high readmission rates  [42]. Thus, although the limited performance achieved by the prediction models prevents establishing associations, these or future analyses could generate hypothesis for further investigation. Our dataset is extracted from a single NHS hospital and our final patient cohort was subject to strict inclusion criteria meaning our results are not generalisable to all inpatients. Restricting our cohort to patients with an inpatient stay of >3 days excluded the most acutely unwell and fittest patients, truncating the range of physiological and metabolic variation and perhaps excluding those whose outcome was easiest to predict.
MVTS data were limited to commonly measured variables and we were limited to broad categorisation of some features, such as the highest level of ICD-10 categorisation. Considering our modelling, as noted above, the HMM has strong and unrealistic assumptions and the clinical interpretations and classifications provided require careful consideration. Our prediction work was also limited to exploratory analyses and would benefit from inclusion of additional patient features and use of ML techniques that account for the sequential nature of data. Lastly, although the discriminative learning methods performed reasonably well, similar to other EHR schemes, it is important to realise the relatively poor precision. This should be expected since the goal of healthcare is to change discharge status via a variety of means, but it also emphasises the difference between providing useful summaries of patient data consistent with previous knowledge for decision support, compared to making a hard 'call' of outcome.
Nevertheless, we generated a research ready dataset using hospital EHR data that was appropriate for both generative and discriminative ML techniques. The clinical interpretation of the HMM states was novel, with previous attempts limited to interpretation of states as disease stages  [19, 43, 44] , and promoted knowledge sharing between clinicians and data scientists. Additionally, the HMM appeared to capture real biological signal. Not only were all 17 states interpreted by the clinician and associated with hold-out clinical information, but generative and discriminative models converged on outputs which made clinical sense, such as associations between inflammation and mortality. Both approaches appeared complementary, discriminative analyses using the raw MVTS variables achieved higher prediction and generative modelling facilitated clinical interpretation, and both can provide a landscape for hypothesis generation.
ROC-AUC (and weighted ROC-AUC) results for the models with highest performance on the prediction of inpatient mortality, 30-day clinical outcome, diagnosis at admission and diagnoses at discharge. Multivariate time series; * Same performance with both input datasetsAll modelling requires assumptions and decisions about a variety of topics before the application of methods, including: what data to collect? what regularisation in time and distribution to use? how should missing data be handled?.
Acknowledgements & Funding This research was supported by the  NIHR Cambridge Biomedical Research Centre  ( BRC-1215-20014 ). The views expressed are those of the authors and not necessarily those of the  NIHR  or the  Department of Health and Social Care. VLK was funded by a  MRC/NIHR Clinical Academic Research Partnership  Grant (CARP; grant code:  MR/T023902/1 ). VT is supported by  Cancer Research UK. EB and TF were funded by the  EMBL European Bioinformatics Institute  ( EMBL-EBI ).
Results from prediction models focusing on first days of admission We used straightforward discriminative models on this dataset and enumerated a relatively large number of hypotheses (16 x 6 = 96), defined as combinations of: a) prediction model (Random Forests, RF or Logistic Regression, LR), b) input data representation (MVTS or HMM states), c) inclusion or not of phenotypic information (i. e. , primary diagnosis at admission), d) different input days lengths, and e) different predicted clinical outcomes (Supplementary Table  4 ). We registered these hypotheses before running tests at the Open Science Framework. For each outcome, the model with the highest performance by 'group' (defined as the combination of model, input data representation and phenotypic information) was identified. We refer to these several combinations as 'models' for simplicity.
When the best performing model employing the HMM states representation of physiological variables was examined, demographics and diagnosis codes were the main drivers of PDR prediction (age, sex, frailty score and ICD-10 code 'S'). The prediction of primary diagnosis at admission (PDA) differs for the different diagnosis codes. An analysis of F1-score by class showed that prediction performance was usually higher for diagnosis codes with more instances in the dataset, ICD-10 codes J, K, N, I, S, C, with the exception of D (good performance with small number of instances) and A (worse performance than other codes with similar numbers). Prediction of diagnosis at discharge (DD) was defined as a multilabel classification problem because one patient can be predicted to have several diagnosis codes.
DISCUSSION AND CONCLUSIONS We have shown a practical way to provide a dataset from an NHS hospital EHR, which is appropriate for ML using the well-established generative time-dependent HMM framework and discriminative RF and LR models. Important aspects of our work were development of the governance structure to facilitate analysis of routinely collected hospital EHR data by technical experts, and development of a protocol to regularise and impute this data into research ready components. We incorporated sufficient anonymisation processes to safeguard privacy, whilst leaving enough information for hidden relationships within the data to be explored and employed secure mechanisms for data transfer and access. We believe it will be critical to store these specific datasets as they are generated, as well as protocols for specific transformations as defined outputs, to progress this field and generate both reproducible research and share expertise.
AEs were included if patients were admitted to a hospital ward (i. e. , not discharged from the Emergency Department) and if information on disease phenotype on admission (admission diagnosis) or discharge was available. Only the first AE within the study period for each Multiple Linear Imputation Interpolation Data cleaning, normalization and regularization Rich-information days ≥ 4 Vital Signs + ≥ 14 Lab Tests Dataset Imputed Preliminary Dataset and Imputed Final Regular Initial Irregular Dataset Regularized dataset with missing observations Poor-information days <14 Lab Tests + < 4 Vital Signs.
EARLY INFLAMMATORY RESPONSE-LIKE STATE AEs grouped by Primary Diagnosis AEs grouped by Primary Diagnosis AEs grouped by Inpatient Mortality Inpatient Death (ID) or Discharged Alive (DA) AEs grouped by Inpatient Mortality Distribution of Lab Tests and Vital Signs Distribution of Lab Tests and Vital Signs Markedly high: wbc, neutrophils; High: monocytes, urea; Relatively high: pulse, resp, liver Not associated with: Diagnostic codes Associated with: Discharged as deceased; function tests, creatinine Slightly high: temp Tends to be present at beginning and resolve Mildly low: hgb, hct, eosinophils, by discharge if outcome is discharged as alive. lymph, diastolic and systolic BP ID DA Figure 8. Correlation between mean values of laboratory test results and vital signs variables in the 'training and test' and 'hold-out validation' datasets.
Literature Search and Inclusion/Exclusion Criteria A literature search was conducted to identify publications related to occupational cancer in Iran through three bibliographic databases (PubMed, Web of Science, and Google Scholar) from 1973 to January 2021. Select keywords were searched through title, abstracts, and body text to identify relevant publications. The search queries (Tables  S1  and S2 ) included keywords according to the PECOs statement (Population: workers, and Iran, Exposure: occupational carcinogenic exposures, Comparison: unexposed/exposed workers, Outcomes: cancer)  [7]. The search strategy combined the above terms by Boolean search operations (AND, OR, NOT) and Mesh terms. We included full publications of epidemiological studies (cohort and case-control studies on occupational cancer) and exposure monitoring studies in which the authors projected the subsequent risk of cancer.
Data Extraction Identified articles were imported into EndNote reference manager, and duplicates were removed. Two reviewers (B. H. & F. M. O. ) independently screened the articles according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines  [8]. Titles and abstracts of all identified references in the primary search were screened to determine potential eligibility. In any case of disagreement between reviewers, the full-text publication was reviewed and discussed to resolve discrepancy; a third reviewer (A. O. ) was consulted to reach consensus if needed. Following the primary screening, full texts of the relevant references were obtained. Five full-text articles had to be requested from authors via E-mail or ResearchGate, and all were successful.
The following data were extracted from each retained publication: name of author, publication year, geographical region of Iran (North/South/West/East/Centre/Tehran), study design (Cohort/Case-control/Exposure monitoring with cancer risk projection as an outcome/burden of cancer studies with nonspecific study design, e. g. , cancer projection using available data on exposure and outcomes), occupation (Title), industry (Title), exposure agents (Names), population size/sample size (Number of workers/samples), exposure assessment methods (Personal air monitoring/Biological monitoring/Stationary air monitoring/Self-report of job or exposure history/Registries/Expert Assessment), and study outcomes (Overall and specific cancer sites).
Results The publication selection process is summarized in the modified PRISMA flow diagram in Figure  1. The initial systematic searches identified 1255 publications (718 from PubMed, 391 from Web of Science, 146 from Google scholar). After removing the duplicates (n = 279), 976 publications remained. Of these, 835 were excluded by screening the titles and abstracts (in vitro and in vivo, nutritional studies, editorial, case reports, reviews and systematic reviews, physical and psychosocial exposure in the workplaces). The remaining 141 full-text publications were reviewed, and 92 were additionally excluded, i. e. , studies evaluating occupational exposures in relation to health outcomes other than cancer. In total, 49 relevant publications (one cohort study, 11 case-control studies, 34 exposure monitoring studies with cancer risk projection as an outcome, and three cancer burden studies using national and international data) were retained for data extraction.
Burden of Cancer Studies In addition, there were three relevant publications that used secondary data to project occupational cancer risk  [55] [56] [57]. Of these, two publications by Mosavi-Jarrahi et al. projected the fraction of leukemia and lung cancer incidence in Iran attributable to exposure to occupational carcinogens, e. g. , benzene, ionizing radiation, and ethylene oxide for leukemia, and silica, cadmium, nickel, arsenic, chromium, diesel fumes, beryllium, and asbestos for lung cancer  [55, 56]. Both publications applied available country-specific workforce survey data from the International Labour Organization (ILO) and estimated the proportion of exposed workers in each industry using the European CARcinogen Exposure (CAREX) database and relative risk estimates from the literature to predict the burden of occupational cancer  [58].