dataesr/scientific-paragraphs-categorization huggingface dataset:
For the case with SVD- SPPMI derived embeddings, one may only release higher dimensional vectors since the lower dimensional embeddings can be recovered via simple truncation. However, we do not recommend using higher dimensions for all tasks. In this paper, we derived embeddings via SVD-SPPMI, considered in the literature as equivalent to the skip-gram algorithm with negative sampling (SGNS)  17. Computationally, the SVD-SPPMI approach is substantially more efficient than SGNS as it does not need to conduct the negative sampling which is computationally intensive especially when the number of codes is massive. Due to both IRB and computational constraints, we are only able to derive embeddings from SVD-SPPMI and GloVe which only require summary data and but not SGNS. We find that SVD-SPPMI derived embeddings generally have more robust performance compared to those from GloVe which also appears to suffer from convergence issues when fitting for higher dimensions, possibly due to the sparsity of the SPPMI matrices.
The embedding vectors provide not only a method to share and analyze data, but also an opportunity to develop an integrated clinical knowledge network with input from many institutions. This network allows us to visualize the node-wise relationships between a target code (e. g. , a PheCode or a RxNorm) and its neighborhood codes: PheCode, RxNorm, CCS, and Labs (Fig. 4 ). By leveraging information from both sites, the integrative network covers all available knowledge and consists of a more comprehensive pool of neighborhood codes compared with local networks. Finally, using KESER, this knowledge network can be updated over time to study relationships between emerging conditions and their relationships with existing conditions, across multiple healthcare systems. This is particularly relevant for future studies on the impact of the COVID-19 pandemic.
Data pre-processing The VA Corporate Data Warehouse (CDW) aggregates EHR data from over 150 VA facilities into a single data warehouse. It contains clinical, financial, and administrative records for over 23 million unique individuals (1999-2019). The CDW supports both business operations and research. A total of 12. 6 million patients with inpatient and outpatient codified data from at least one visit were included for this analysis. We defined outpatient visits to include services from all VA outpatient stop codes. There are over 500 outpatient stop codes that cover a wide range of services such as emergency department visits, therapy, and primary care. We first extracted records from the CDW. We then grouped each patient's records together in ascending chronological order. Codes occurring multiple times for the same patient within the same day are counted once per day.
The resulting files were stored using parquet, a columnar storage format. The parquet file format was well suited to storing this data compactly while also allowing parallel processing. Mass General Brigham (MGB), formerly Partners Healthcare, is a Bostonbased non-profit healthcare system anchored by two tertiary care centers, Brigham and Women's Hospital (BWH) and Massachusetts General Hospital (MGH). The Research Patient Data Registry (RPDR) of MGB is a research copy of the EHRs of BWH and MGH with over 1 billion visits containing diagnoses, medications, procedures, and laboratories information. The patient population included 2. 5 million patients with at least 3 visits spanning more than 30 days. The analysis included coded data from all inpatient, outpatient, and emergency department visits between 1998 and 2018.
We used the same format as VA described above to store patient visit level data for processing. We gathered four domains of codified data including diagnosis, procedures, lab measurements, and medications from VA and MGB EHRs. Since multiple EHR codes can represent the same broad concept, (e. g. , acute myocardial infarction (MI) of anterolateral wall and acute MI of the inferolateral wall are separate codes that describe the same concept of MI), we rolled individual codes to a code representing a general concept. ICD codes were aggregated into PheCodes to represent more general diagnoses, e. g. , MI rather than acute MI of inferolateral wall, using the ICD-to-PheCode mapping from PheWAS catalog (https://phewascatalog. org/phecodes). We utilized multiple levels of granularity of PheCode, including integer level, one-digit level and two-digit level.
Creating embedding vectors We obtained embeddings by performing SVD on the SPPMI matrix, known as the SVD-SPPMI algorithm. This approach provided embeddings considered as efficient and equivalent to those derived from the skipgram algorithm with negative sampling  17, 20, 30, 31. We first constructed code co-occurrence matrices as described in Beam et al. 14. For any given patient, we scanned through each of their codes as a target code. For any given target code occurring at time t, denoted by w t , we counted all codes occurring within 30 days of t as co-occurrences with w t. The total numbers of co-occurrences for all possible pairs of codes are aggregated over all target codes within each patient and then across all patients, yielding the co-occurrence matrix, denoted by C ¼ C w; c ð Þ ½.
Although only codes that occur after the target code are considered, this is the same as finding co-occurring codes within 30 days of the target code (i. e. , between -30 and 30 days), owing to the symmetry of the data. Thus, given a target phenotype w (e. g. , PheCode 714. 1 for RA), we assume the context codes vocabulary V C ðwÞ are the codes co-occurred with the target word within a 30-day window. This step requires considerable computational resources and a detailed algorithm for efficiently computing the co-occurrence matrix was created for this study (https://github. com/rusheniii/LargeScaleClinicalEmbedding). Since our sparse regression procedures (described in later sections) require selection of tuning parameters, we constructed two separate cooccurrence matrices at each site. At VA, from the 12.
Knowledge extraction via sparse embedding regression (KESER) We propose to infer conditional dependency among the clinical codes based on the conditional dependency among their corresponding embedding vectors. To provide a rationale for this framework, we note that the skip-gram model with negative sampling  16  directly encodes the marginal dependency between the target code w and its context code c via the covariance between their respective embedding vectors V w and V c with logit P Target code w and context code c co À occurs w; c j ð Þ ¼ V T w V c : (2) This motivates us to model the conditional distribution of the target code w and other codes by imposing a Gaussian distribution on the embedding vectors and inferring the dependency via a Gaussian graphical model on top of the skip-gram model.
Due to computational constraints, we performed sensitivity analyses to evaluate how window size and k impact the embedding quality using the MGB Biobank consisting of EHR data from about 70 K patients. We derived embeddings with co-occurrence matrices constructed with window sizes ranging from 7, 30 up to 60-days, and k ranging from 1, 5, to 10. To select dimension d, we first initialized the dimensions by retaining 95% of the variation in the SVD, denoted by d 95%. Subsequently, we considered two data-driven strategies for optimizing the dimension up to d 95% by maximizing (i) the signal to nose ratio (SNR); and (ii) the AUC, where SNR d ð Þ ¼ W d =S d , W d and S d are the average cosine similarity among all pairs with known relationships and among all random pairs. For similarity, we used the PheCode hierarchy for tuning optimal dimensions and defined pairs as similar if they shared the same integer to calculate the SNR and AUC.
For relatedness, we used 10% of the known related PheCode-PheCode pairs from Wikipedia and PheCode-RxNorm pairs from https://www. drugs. com/ and MEDRT to tune the dimension and used the remaining known related pairs for validation. We evaluated the quality of the derived embedding vectors by quantifying their accuracy in detecting known similar pairs (RxNorm-RxNorm and Lab-Lab) and related pairs (PheCode-PheCode, PheCode-RxNorm), and evaluated the KESER algorithm by quantifying its power in detecting known related pairs as described above. For each type of relation, since a vast majority of pairs are unrelated, we randomly sampled a large number of pairs within each type of relationship to obtain the reference distribution for unrelated pairs. For each type of relationship, we obtained the cosine similarity of the embedding vectors between known pairs and between random pairs.
We first calculated the area under the AUC as an overall accuracy summary. We then reported the sensitivity of detecting related pairs by thresholding cosine similarities to achieve a FPR of 0. 01, 0. 05 or 0. 10. We also evaluated the performance of the KESER for feature selection at each site and integrative feature selection at both sites. We report the sensitivities in detecting known related PheCode-PheCode and PheCode-RxNorm pairs, that is the proportion of pairs detected by KESER among all known pairs. The trained embeddings at MGB and VA can be used to map codes across the two institutions via orthogonal transformation similar to language translation  26. Specifically, let V ðVAÞ and V ðMGBÞ denote embedding vectors for codes that are common to both institutions. We may find an orthogonal matrix Q to minimize the distance between V ðMGBÞ and V ðVAÞ Q as in Smith et al.
26. We used 1823 codes (223 CCS, 178 LOINC, 698 PheCode, and 724 RXNORM) that are common to MGB and VA to train Q. The test set consists of 1000 PheCodes that are common to both institutions but not included in the training set as well as a set manually curated 251 VA local medication code → RXNORM mappings. We evaluate the quality of the cross-institution mapping based on the top-1, top-5, and top-10 accuracy calculated based on the test set. We performed the codemapping with embeddings of dimensions chosen both via AUC and SNR. Patients with RA are treated with disease-modifying anti-rheumatic drugs (DMARDs), treatments that can prevent the progression of RA. A list of 16 RA treatments approved prior to 2017 were manually curated by domain experts and grouped into two categories: (1) DMARDs currently in use, (2) RA-related drugs used in conjunction with DMARDs.
We reported sensitivities in detecting the RA-related drugs using KESER against this manually curated list. One downstream application of feature selection is to develop supervised phenotyping algorithms for classifying disease status with these selected features. Supervised algorithms are typically developed using a training dataset consisting of gold standard labels and observations on a given set of candidate features  34. Existing phenotyping algorithms have considered various approaches to selecting candidate features including the main PheCode of the disease only (PheCode), all features (FULL), or informative features selected manually or extracted using unsupervised algorithms such as SAFE  15. Using codified EHR data from 68,213 MGB Biobank participants, we compared the performance of supervised phenotype algorithms trained with these existing feature selection strategies to those trained with KESER-selected features.
ACKNOWLEDGEMENTS We would like to acknowledge that this work would not have been possible without the collaboration between VA and the  Department of Energy  which provided the computing infrastructure necessary to develop and test these approaches at scale using nationwide VA EHR data. We would also like to thank  Ms. Hope Cook , SQL Database Administrator, and  Mr. Ian Goethert , Data Engineer, for their assistance with optimizing the computing environment required for this study. This manuscript has been in part co-authored by  UT-Battelle, LLC  under Contract No. DE-AC05-00OR22725 , and under a joint program with the  Department of Veterans Affairs  under the  Million Veteran Project Computational Health Analytics for Medical Precision  to Improve Outcomes Now. Part of this research is based on data from the  Million Veteran Program, Office of Research and Development, Veterans Health Administration , and was supported by award # MVP000.
Another common difficulty of signature attribution packages is the low reproducibility as well as scalability of computations in high-performance computing environments. This is a particular concern when analysis is performed on large datasets, and when the number of bootstrap variations is considerable. We resolve this issue by utilising Nextflow  [10] , a domain-specific language (DSL) designed to primarily address computational irreproducibility and efficient parallel execution in a large number of computing environments, from individual workstations to server clusters and cloud computing services.
nnls function in scipy library in Python), however, such approach is known to over-fit the data and lead to a large number of false-positive findings, as shown using simulations below. To mitigate this, ad-hoc approaches can be utilised, such as the optimisation of signature attribution using additional penalties on each signature's contribution, or by using pre-existing biological knowledge such as strand bias rules  [2, 12]. Although such optimisation is implemented in MSA using the penalty loops described below, the main advantage of the tool is the ability to quantify the confidence of attribution for each signature.
Parametric bootstrap using Multinomial distribution Parametric bootstrap assumes that the data follow a known underlying distribution. This implies making certain assumptions on the original dataset, so that the bootstrap samples can be drawn from the estimated parametric model. Here, we make an assumption that mutations are accumulated following Poisson distributions for each mutation class, such as a specific trinucleotide context, i. e. that in each class, mutations accumulate randomly, independently and at a constant rate. The idea of parametric bootstrap applied to mutational processes was inspired by mut-Signatures package  [9]  (which is itself based on the original MATLAB framework for deciphering signatures  [16] ), where the input mutation matrix is bootstrapped according to the multinomial distribution Multinomial(M, p 1 ,.
Automation and integration The MSA tool is implemented as a set of scripts in Python language, with simulation, optimisation and final attribution steps fully automised using Nextflow workflow management system  [10]. All dependencies, including packages such as pandas, numpy, scipy, matplotlib and seaborn, are automatically handled by Nextflow via containerisation using Docker technology, as well as Singularity where Docker is not available. Users not willing to use Docker or Singularity may opt to use a Conda environment that is also automatically handled by Nextflow, yet this approach has inferior reproducibility compared to container technology. Native support of SigProfilerMatrixGenerator  [5]  and SigProfilerExtractor  [6]  tools is implemented in MSA for convenience. Where SigProfiler outputs are available, running MSA is as simple as executing a single line containing the output paths:.
nextflow run https://gitlab. com/s. senkin/MSA -profile docker --dataset test --SP_matrix_generator_output_path path/to/SP_ME/ --SP_extractor_output_path path/to/SP_ extractor/ Mutational classifications currently supported are SBS-96, SBS-192 and SBS-288 for single base substitution signatures, DBS-78 for doublet base substitution signatures, ID-83 for small insertion and deletion signatures, SV-32 for structural variant signatures. The SV signatures are implemented as an experimental feature.
Validation with simulations When simulating a dataset that is supposed to resemble real data, one generally has to make certain assumptions about underlying distributions. Firstly, we explored scenarios mimicking real cohorts (e. g. shown on Fig. 1 ), with a defined set of acting reference COSMIC signatures, where generated signature attributions follow non-negative zero-inflated Gaussian distributions. Generally, such distributions do not describe real attributions well, especially for rare signatures. Therefore, as a default approach fully automised within the MSA Nextflow pipeline, we use data-driven simulations based on a simple bootstrap of signature activities derived without regularisation (i. e. with zero penalties). These activities are used to generate mutational profiles which are then resampled with replacement and injected with noise.
(a) (b) Fig. 3  Overall performance metrics of signature attribution for simulated ESCC cohort with respect to optimisation penalty. a Senstivity, specificity and Matthews correlation coefficient are calculated using point estimates without confidence intervals. b Similar metrics are calculated using bootstrap variations yielding confidence intervals. Red horizontal line corresponds to the 95% value for each metric Finally, performance of MSA was benchmarked against existing tools using both real and synthetic data published by the PCAWG consortium  [19]  (Additional file 1), as well as simulations based on the real PCAWG data, for SBS, DBS and Indel mutation types (Additional file 2). MSA generally achieves higher attribution performance than that of other tools due to its automised optimisation of the penalties applied, coupled with the application of confidence intervals.
, p m ) are taken from real data which can be inaccurate. However, this method aims to estimate the statistical uncertainty of signature attribution method rather than total uncertainty of attribution, and for such purposes remains adequate. Fig. 4  Performance metrics for COSMIC signatures SBS5 and SBS16 with respect to optimisation penalty. SBS5 metrics are derived using point estimates (a) and confidence intervals (b). Similarly, SBS16 metrics are derived using point estimates (c) and confidence intervals (d). Red horizontal line corresponds to the 95% value for each metric Parametric bootstrap can be applied using both simple NNLS attribution and the one based on penalised optimisation designed to maximise sensitivity and specificity of signature attribution. We developed a set of tools assisting in the validation of optimisation parameters for any given scenario using automised data-driven simulations of different cancer types and population cohorts.
The tool comprises a set of Python scripts unified in a single Nextflow pipeline with containerisation, specifically designed for crossplatform reproducibility and scalability in high-performance computing environments. MSA is publicly available at https:// gitlab. com/s. senkin/ MSA with an extensive documentation at https:// gitlab. com/s. senkin/ MSA/-/ wikis. • fast, convenient online submission • thorough peer review by experienced researchers in your field • rapid publication on acceptance • support for research data, including large and complex data types • gold Open Access which fosters wider collaboration and increased citations maximum visibility for your research: over 100M website views per year.
Gap models are often cohort-based, assuming that all trees of the same species and age behave similarly, for the sake of simulation efficiency. By contrast, ForCEEPS is completely individual-based, which notably allows to simulate the intraspecific variability in competitive ability. Another novel aspect in ForCEEPS is the possibility of imposing a feedback between the actual forest composition and the identity of the colonizing seedlings each year. This latter feature may be crucial for examining mechanisms of species coexistence in tree communities  (Cordonnier et al. 2018b ). However, with regard to the objective of the present paper, the most crucial development of ForCEEPS in comparison with ForClim is the implementation of a new module for tree-tree competition for light, i. e. a key factor controlling growth and forest structure  (Schwinning and Weiner 1998) , where the individual crown lengths are explicitly represented in the vertical canopy space (see Appendix A).
Parameter gs. This parameter is the most difficult to calibrate as it requires data from trees growing in "optimal conditions", which are scarce in observational datasets as the growth of trees is usually constrained by environmental conditions or biotic factors (e. g. competition). To cope with this challenge, we took advantage of the NFI that covers a very wide range of conditions (in both space and time), providing a large number of "annual diameter increment vs. diameter" pairs for each of the 11 species (i. e. , the nine main species and C. betulus and Q. pubescens) for which abundant data were available (n=206,569 for all species confounded, Table  S2 ). For each of these 11 species, we grouped trees according to their diameter (according to 1-cm size classes) and selected the 10% of trees with the greatest annual diameter increment, assuming that these trees grew in "optimal conditions" or at least under almost unconstrained conditions.
However, we note that the annual increments are derived from five-year average, which may lead to an underestimation of the actual greatest annual diameter increments. Then we fitted gs from Eq. 1 with this dataset, using a non-linear least squares approach implemented by the nls function in the R software (R Core Team 2018). For the remaining species (n = 11), the gs values have been set from previous studies  (Didion et al. 2009). The fitted values for the parameters gs ranged from 79 to 399 (Table  S1 ). These values are consistent with former estimates for the same or related species  (Bugmann 1994 , Didion et al. 2009 ). Parameter ss. The calibration of ss (Eq. 2) relied on NFI data because of their representativeness of the conditions in which each species occurs. The whole NFI dataset was used for the calibration to cover the largest range of conditions in which each species occurs.
Although diameter-height relationships were shown to be affected by environmental conditions, e. g. climate, tree social status and stand density  (Trouvé et al. 2015 , Fortin et al. 2019 ), these factors were not accounted for in the model. The rationale for this lies in our aim to keep the model structure as simple as possible to allow for an easy parameterization and use at large scale for a large number of species. We fitted the height-diameter relationships (Eq. 2) on the NFI dataset, using the nls R function, and extracted ss values for each species. As for gs, this calibration was conducted for the 11 main species, while we relied on  Didion et al. (2009)  for the 11 additional species. Parameter. This parameter was calibrated using NFI data and/or literature  (Rameau et al. 1989 (Rameau et al.
Congruence of key parameter values with functional traits. To gain mechanistic insight into the parameters values derived from the calibration procedure, we evaluated the congruence of key model parameters with functional traits extracted from the literature. To do so, we first selected the most meaningful ForCEEPS parameters in terms of species ecological strategies, including gS, DrTolS, ShTolS, ShTol_seedlingS, and NreqS. Then we collected data on relevant traits from various database, including: xylem cavitation resistance (assess through the water potential causing 50% cavitation, Ψ50 in MPa), leaf turgor loss point (Ψtlp, in MPa), water potential causing stomatal closure (Ψclose, in MPa) and safety margins from Ψtlp and Ψclose [from the SurEAu database, Martin-StPaul, Delzon, & Cochard (2017)], wood density (g/m 3 ,  Chave et al.