dataesr/scientific-paragraphs-categorization huggingface dataset:
Apart from SA, which lacks this capability, all tools were also forced to extract the known number of ground-truth signatures. Results from the suggested approach reflect the expected outcome from running a tool on an unknown dataset, while results from the forced approach allow understanding limitations in each tool's implementation. Our evaluation reveals that most tools can successfully extract mutational signatures from easy scenarios with the majority of F 1 scores >0. 90 (Figure  2A ). 36] 38  In contrast, medium scenarios have proven to be a challenge for most tools with only SigProfilerExtractor, SigProfiler_PCAWG, and SA exhibiting F 1 scores >0. 90. All tools had worst performance for the hard scenarios with F 1 scores below 0. 80; only SigProfilerExtractor had an F 1 score of $0.
SigProfilerExtractor and SA are the only two tools that support analysis of custom-size matrices and provide GPU support (Table  1 ), thus allowing analysis of data with an extended number of mutational channels within a reasonable time frame. In contrast, all other matrix factorization tools rely solely on CPU implementations, with full runs expected to take many months for each tool applied to these scenarios. Overall, SigProfilerExtractor outperformed SA with average F 1 scores of 0. 92 and 0. 85, respectively (Table  S2 ). To further compare SigProfilerExtractor with the other seven top-performing tools, we applied each tool to a dataset with 30 ground-truth SBS-96 signatures operative in 1,000 genomes and random noise between 0% and 10%. Analysis for each noise level was repeated 20 times to account for variability in the noise generation.
SigProfilerExtractor, SomaticSignatures, MutSpec, and SignatureToolsLib were robust to noise, with mostly unaffected performance (Figure  3B ; Table  S3 ). In contrast, SigProfiler_PCAWG, SA, SigneR, and MutationalPatterns were susceptible to noise (Figure  3B ). For example, 2. 5% noise reduced SA's F 1 from 0. 76 to 0. 66, while 10% noise reduced its F 1 to 0. 07. Similarly, 10% noise reduced the F 1 of SigProfiler_PCAWG from 0. 71 to 0. 58, the F 1 of SigneR from 0. 61 to 0. 43, and the F 1 of MutationalPatterns from 0. 60 to (A) Average precision (x axes), sensitivities (y axes), and F 1 scores (harmonic mean of precision and sensitivity; red curves) are shown across the three types of scenarios. Different tools are displayed using circles and triangles with different colors. Circles are used to display results for suggested model selection, which most closely matches analysis of a real dataset.
Ratio of F 1 scores (y axes) with standard errors of the mean was calculated for all medium and hard scenarios. Ratio of approximately 1. 00 indicates a similar performance between suggested and forced model selection. (D) Summary of the performance for the top eight tools on hard SBS-96 noiseless scenarios with suggested model selection. Vertical axes reflect F 1 score (left plot), sensitivity (middle plot), and false discovery rate (right plot), respectively. Error bars correspond to standard errors of the mean. Results from SignatureAnalyzer and MutSignatures are not displayed in (A)-(C) for forced and suggested model selections, respectively, as the tools do not support these types of analyses. Cell Genomics 2, 100179, November 9, 2022 7 Technology 0. 37. SA's reduced performance on data with noise is due to its automated approach for selecting total number of signatures.
SA uses automatic relevance determination (ARD)  33  for selecting the number of signatures, with this number increasing from 26 (no noise; 30 ground-truth signatures) to 96 signatures (10% noise; Table  S3 ). In contrast, SigProfiler_PCAWG, SigneR, and MutationalPatterns exhibit similar performance between forced and suggested solutions on data with noise (Table  S3 ), indicating that their reduced performance is likely due to their factorization approaches. SigProfilerExtractor outperformed all other tools regardless of noise levels. Simulations with 5% noise reflect genomics datasets with $0. 95 average sensitivity and precision of single base substitutions, similar to the recently published PCAWG cohort, which has 95% sensitivity (90% confidence interval, 88%-98%) and 95% precision (71%-99%).
43 For WGS simulations with 5% noise, SigProfilerExtractor was able to identify between 20% and 50% more TP signatures while yielding more than 5-fold less FP signatures compared with the next seven best-performing tools (Figure  3C ; Table  S3 ). To assess the ability of the top-performing tools to extract de novo mutational signatures from exome sequencing data, a WES benchmarking dataset, encompassing 20,000 unique synthetic cancer exomes, was generated by downsampling the WGS noise scenario with 5% noise. Exome data were challenging for all the de novo mutational signature extraction tools, resulting in a significant decrease in performance (Figure  3D ). The average F 1 score for all tools dropped from 0. 61 for WGS simulations with 5% noise to 0. 46 for WES simulations with 5% noise.
Reanalysis of 4,643 WGS and 19,184 WES human cancers To demonstrate its ability to yield novel biological results, SigProfilerExtractor was applied to the recently published set of 2,778 WGS cancers from the PCAWG project. 43 Additionally, we applied SigProfilerExtractor to an extended cohort of another 1,865 WGS and 19,184 WES cancers, encompassing data from The Cancer Genome Atlas (TCGA)  49  as well as 261 other published studies and 35 different ICGC projects (Table  S8 ). As previously done in our original PCAWG analysis of mutational signatures,  12  extraction of mutational signatures was performed within each cancer type and across all samples (STAR Methods). In addition to all previously detected signatures,  12  our direct application of SigProfilerExtractor revealed three novel mutational signatures in the PCAWG dataset: SBS92, SBS93, and SBS94.
Technology De novo extraction from the Mutographs cohort of 552 WGS esophageal squamous cell carcinomas, 52 a cancer type not included in the PCAWG dataset,  43  identified an analogous mutational signature, with the similarity extending to both trinucleotide context and transcriptional strand asymmetry (Figure  4C ; cosine similarity: 0. 88; p = 1. 1 3 10 À6 ). Signature SBS94 was found at high levels in a single colorectal PCAWG cancer, with smaller contributions to another eight colorectal cancers. The pattern of SBS94 was characterized by C>A mutations with a strand asymmetry indicative of damage on guanine (Figure  4D ). Validation of somatic mutations by visual inspection confirmed that 98% of mutations contributed by SBS94 are likely real. Signatures SBS93 and SBS94 did not associate with any of the available PCAWG metadata,  43  and their etiologies remain unknown.
DISCUSSION The performed large-scale benchmarking demonstrates that SigProfilerExtractor outperforms 13 other tools for de novo extraction of mutational signatures for noiseless datasets as well as for datasets containing different levels of random noise, including synthetic data emulating WGS and WES cancers. Importantly, SigProfilerExtractor generates almost no FP signatures while still identifying a higher number of TP signatures when compared with any of the other tools (Figures  2D, 3C,  and 3D ). De novo extraction relies both on a factorization approach and on a model-selection algorithm for determining the total number of operative signatures (Figure  1 ). Benchmarking with forced model selection, where tools were required to extract the known number of ground-truth signatures, Mutational signatures are displayed using 96 plots.
Similarly, benchmarking with suggested model selection, which most closely matches analysis of a real dataset, further demonstrates SigProfilerExtractor's ability to reveal novel biological results (Figure  2A ; Tables  S1, S2 , S3, S4, and S5). Interestingly, SigProfilerExtractor outperforms other tools when extracting correlated mutational signatures  53  and signatures with overlapping profiles for specific contexts. In scenarios 5, 6, 9, and 10 (based on COSMIC signatures SBS2, SBS7a, and SBS7b, which share specific subtypes of C>T mutations), SigProfilerExtractor exhibited an average F 1 score of 0. 96, while the next best tools had F 1 scores <0. 90 (Table  S1 ). While our benchmarking evaluated 13 additional tools, 6 of the 13 tools internally rely on the same computational engine. Maftools, MutationalPatterns, MutSpec, SignatureToolsLib, SigMiner, and SomaticSignatures use the NMF R package  23  to perform their factorization (Table  1 ), albeit with slightly different hyperparameters and, in some cases, distinct pre-processing of the input matrix.
Predictably, these six tools have similar performance across many of the scenarios (Tables S1, S2, S3, S4, and S5). SigProfiler_PCAWG and MutSignatures utilize similar implementations of NMF. 26  TensorSignatures makes use of the standard factorization algorithms included in Tensor-Flow. 41 SigFit uses a previously developed nonnegative factorization method. 29 In contrast, EMu, SA, SigneR, and SigProfilerExtractor provide original implementations of their factorization algorithms (Table  1 ). EMu was originally developed and tested on small datasets,  20  and its benchmarking performance is perhaps unsurprising considering the large number of synthetic samples used in all scenarios. Surprisingly, the original implementations of SA and SigneR were susceptible to noise, yielding high numbers of FP signatures (Figure  3B ).
Seven of the tools did not provide an automatic approach for selecting the total number of signatures (Table  1 ). Instead, most of these tools offered methodologies for manual selection, thus, bringing user dependence and arbitrariness in selecting solutions. EMu, TensorSignatures, and SigneR automatically select the total number of signatures using Bayesian information criterion (BIC),  21  while SA and SigMiner utilize ARD. 33 SigFit's selection approach is based on the Elbow method. 30 SigProfilerExtractor leverages a modified version of the NMFk selection approach. 37 Importantly, our simulations demonstrate that SigProfilerExtractor's model selection is robust to noise, while the implemented BIC and ARD approaches are affected even by low levels of noise (Figure  3B ). High noise levels had limited effect on SigProfilerExtractor, causing the tool to miss some of the ground-truth signatures used to generate the synthetic datasets.
In all cases, average values exceeded 0. 88, indicating that mostly flat signatures are erroneously called by these tools (Table  S4 ). In the case of SigProfilerExtractor, the average Shannon equitability of FP without noise and for all noise levels was, in all cases, over 0. 92, following a similar trend as the previously mentioned tools. However, it is worth noting that only three FP signatures were detected by SigProfilerExtractor in all 20 replicates with 10% noise (600 total ground-truth signatures), whereas, for example, 417 and 1,819 were found for SigneR and SA, respectively (Table  S3 ). In addition to outperforming 13 other tools on simulated datasets, SigProfilerExtractor can reveal additional biological results, as demonstrated by identifying four novel signatures from the reanalysis of 23,827 sequenced cancers from the PCAWG and the extended datasets.
Data and code availability Our study analyzes synthetically generated data, as well as publicly available data from human subjects. The accessions numbers for the human datasets are listed in the key resources table and Table  S8 , and correspond to a total of 263 published studies as well as 35 ICGC projects. 6, 43, 49, 51,  All r ults from the benchmarking with synthetic datasets, including the appropriate input used to run each of the tools as well as the output generated by each of the tools, can be found at: ftp://alexandrovlab-ftp. ucsd. edu/pub/publications/Islam_et_al_Sig ProfilerExtractor/Benchmark/ and Figshare: https://doi. org/10. 6084/m9. figshare. 20409430. All results from the benchmarking of the different options available in SigProfilerExtractor for matrix normalization, NMF initialization, and NMF objective function can be found at: ftp://alexandrovlab-ftp.
ucsd. edu/pub/publications/Islam_et_al_SigProfilerExtractor/ Benchmark_Initialization_Normalization_Objective-Function/ and Figshare: https://doi. org/10. 6084/m9. figshare. 20411483. All results from the de novo extraction of mutational signatures from the Pan-Cancer Analysis of Whole Genomes (PCAWG) dataset can be found at: ftp://alexandrovlab-ftp. ucsd. edu/pub/publications/Islam_et_al_SigProfilerExtractor/PCAWG_Reanalysis/ and Figshare: https://doi. org/10. 6084/m9. figshare. 20406279. All results from the de novo extraction of mutational signatures from the extended dataset can be found at: ftp://alexandrovlab-ftp. ucsd. edu/pub/publications/Islam_et_al_SigProfilerExtractor/Extended_Cohort_Reanalysis/ and Figshare: https://doi. org/10. 6084/ m9. figshare. 20406326. A summarized collection of all input mutational matrices, as well as de novo extracted mutational signatures and activities for both PCAWG and extended datasets has also been deposited at Figshare: https://doi.
org/10. 6084/m9. figshare. 20293890. All results from the de novo extraction of mutational signatures for confirming the patterns of the novel signatures for additional datasets can be found at: ftp://alexandrovlab-ftp. ucsd. edu/pub/publications/Islam_et_al_SigProfilerExtractor/Confirmation_of_ Novel_Signatures/ and Figshare: https://doi. org/10. 6084/m9. figshare. 20406156.
Technology All results from the de novo extraction of mutational signatures from downsampling of whole-genome sequenced samples to whole-exomes can be found at: ftp://alexandrovlab-ftp. ucsd. edu/pub/publications/Islam_et_al_SigProfilerExtractor/ Downsampling_of_whole_genomes/ and Figshare: https://doi. org/10. 6084/m9. figshare. 20406276. All original code has been deposited at GitHub (https://github. com/AlexandrovLab/SigProfilerExtractor and https://github. com/ AlexandrovLab/SigProfilerExtractorR), PyPI (https://pypi. org/project/SigProfilerExtractor/) and Zenodo (https://doi. org/10. 5281/ zenodo. 6746540 and https://doi. org/10. 5281/zenodo. 6941779). SigProfilerExtractor and all its modules are open source and freely available for use under the permissive 2-clause BSD license. SigProfilerExtractor and its modules are implemented in Python with an R wrapper package allowing users to run the tool from an R environment.
The R version of the tool can be downloaded from https:// github. com/AlexandrovLab/SigProfilerExtractorR. A detailed wiki page including installation, usage, and explanation of results is provided at https://osf. io/t6j7u/wiki/home/. SigProfilerExtractor is compatible with Windows, Linux, Unix, and macOS operating systems. Any additional information required to reanalyze the data reported in this paper is available from the lead contact upon request.
The mutational matrix can be provided as a text file with the first column containing the names of the mutational channels and the first row containing the names of the examined samples, thus supporting nonnegative matrix factorization for any custom matrix dataset. Alternatively, users can provide a mutational catalog of somatic mutations in a commonly used format (e. g. , VCF, MAF, etc. ) and this mutational catalog will be internally converted into the appropriate mutational matrix by SigProfilerMatrixGenerator. 15 odule 2: Resampling of the input mutational matrix and normalizing the resampled matrix SigProfilerExtractor does not factorize the original input matrix. Rather, prior to performing matrix factorization, SigProfilerExtractor performs independent Poisson resampling of the original matrix for each replicate.
Finally, the activity matrix is constructed by combining the activity vectors generated for all samples in the dataset. Module 7: Outputting and plotting of analysis results All previous modules make use of Module 7 for outputting and plotting of the generated results. It should be noted that SigProfilerExtractor provides extensive output for the interrogated total number of operative signatures, k, with a default range of k between 1 and 25 signatures. For each value of k, SigProfilerExtractor outputs the set of operative de novo mutational signatures, the activities of the operative signatures, and an extensive set of information related to individual samples, individual de novo signatures, and the overall convergence of the factorization and clustering. Module 7 also provides additional information when run in debug mode.
Analysis of the genomics data from cancer and normal somatic tissues For all examined whole-genome sequenced cancer and normal somatic tissues, de novo extraction of mutational signatures was performed with SigProfilerExtractor with default parameters using two distinct mutational classifications: SBS-96 and SBS-288. Only the SBS-96 classification was used for whole-exome sequenced data. The SBS-96 mutation classification incorporates the six types of single base substitutions: C>A, C>G, C>T, T>A, T>C, and T>G. Each type of single base substitution is further separated into 16 subtypes determined by the four possible bases 5 0 and -3 0 adjacent to each mutated base. The SBS-288 mutation classification extends the SBS-96 mutation classification by adding additional information for each of the 96 subtypes.
Specifically, SBS-288 incorporates whether a single base substitution is in non-transcribed/intergenic DNA, on the transcribed strand of a gene, or on the untranscribed strand of the gene. De novo extraction was performed separately for all examined datasets. Specifically, SigProfilerExtractor was applied: (i) to all 2,778 whole-genome sequenced cancers from the Pan-Cancer Analysis of Whole Genomes project  43  ; (ii) to all 1,865 whole-genome and 19,184 whole-exome sequenced cancers from the extended cohort (Table  S8 ); (iii) to all samples in each of the 37 cancer types of the Pan-Cancer Analysis of Whole Genomes project  43  with each cancer type examined separately; (iv) to all samples in each of the 66 cancer types of the extended cohort (Table  S8 ) with each cancer type examined separately; (v) to all 88 whole-genome sequenced microbiopsies of histologically normal urothelium 50 ; (vi) to the complete set of whole-genome sequenced bladder cancers from TCGA  56  ; (vii) to exome downsampling of all bladder whole-genome sequenced Cell Genomics 2, 100179, November 9, 2022 e5 Technology cancers from the Pan-Cancer Analysis of Whole Genomes project  43  ; (viii) to exome downsampling of all 88 whole-genome sequenced microbiopsies of histologically normal urothelium.
Creation of scenarios with synthetic datasets Benchmarking was performed on simulated datasets with and without noise. These synthetic datasets were created using a previously described method. 12 All datasets without noise were categorized as different scenarios with many of these scenarios attempting to emulate a particular set of cancer types. Specifically, 20 scenarios were created for the SBS-96 mutational classification and 12 additional scenarios were generated for the extended number of channels. For the SBS-96 classification, COSMIC signatures originally extracted from the PCAWG dataset  12  as well as signatures extracted using SignatureAnalyzer  12  and random signatures were used as ground-truth signatures. Many of the scenarios were created using a combination of tissue specific signatures.
Synthetic scenarios were labeled as easy, medium, and hard based on the number of operative signatures in each scenario. Based on our most recent analysis of mutational signatures in 82 cancer types,  12  approximately 7. 4% of human cancer types have 5 or less signatures (reflected in simulations of easy scenarios), 15. 9% have 11 to 21 signatures (medium scenarios), and 59. 5% have 25 or more signatures (hard scenarios). Note that 17. 2% of cancer types have either between 5 and 10 signatures or between 22 and 24 signatures. Detailed description of each of the used scenarios for benchmarking is provided below. Note that some of the generated scenarios were initially created as part of Ref. 12. The computational approach for generating the synthetic data can be found at: https://github. com/steverozen/SynSigGen.
Noiseless scenarios 1 to 14 had only a single replicate while scenario 15 through 20 had 10 replicates each, and WGS and WES noise scenarios had 20 replicates each. Scenarios 1, 2, E-1, and E-2 The scenarios were generated to emulate a subset of the pancreatic adenocarcinoma PCAWG dataset with a total 1,000 synthetic samples. Ground-truth signatures were based on COSMIC as well as on signatures extracted by SignatureAnalyzer. Scenarios 3, 4, E-3, and E-4 Mutational spectra generated from combinations of flat, relatively featureless mutational signatures. A total of 1,000 synthetic tumors emulating a mixture of 500 synthetic renal cell carcinomas (high prevalence and mutation load from SBS5 and SBS40 signatures) and e6 Cell Genomics 2, 100179, November 9, 2022 Technology 500 synthetic ovarian adenocarcinomas (high prevalence of and mutation load from SBS3).
Ground-truth signatures were based on COSMIC and signatures extracted by SignatureAnalyzer. This dataset embodies tumors with high prevalence of the main flat signatures, SBS3, SBS5, and SBS40, in a realistic context. Scenarios 5, 6, E-5, and E-6 Mutational spectra generated from signatures with overlapping and potentially interfering profiles. A total of 1,000 synthetic tumors composed mostly from SBS2, SBS7a, and SBS7b. Mutational load distributions were drawn from bladder transitional cell carcinoma (SBS2) and skin melanoma (SBS7a, SBS7b). Ground-truth signatures were based on COSMIC and signatures extracted by SignatureAnalyzer. Most spectra contain both signatures SBS7a and SBS7b. The potential interference is between SBS2 (mainly C>T) and SBS7a, SBS7b (mainly C>T). Scenarios 7, 8, E-7, and E-8 Mutational spectra generated from combinations of flat, relatively featureless mutational signatures.
A total of 1,000 synthetic tumors emulating a mixture of 500 synthetic renal cell carcinomas (high prevalence and mutation load from SBS5 and SBS40 signatures) and 500 synthetic ovarian adenocarcinomas (high prevalence and mutation load from SBS3). Ground-truth signatures were based on COSMIC and signatures extracted by SignatureAnalyzer. This dataset embodies tumors with high prevalence of the main flat signatures, SBS3, SBS5, and SBS40, in a simplified fashion, where only these three signatures are present. Scenarios 9, 10, E-9, and E-10 Mutational spectra generated from signatures with overlapping and potentially interfering profiles. A total of 1,000 synthetic tumors composed from SBS2, SBS7a, and SBS7b. Mutational load distributions were drawn from bladder transitional cell carcinoma (SBS2) and skin melanoma (SBS7a, SBS7b).
Ground-truth signatures were based on COSMIC and signatures extracted by SignatureAnalyzer. Most spectra contain both groups of signatures. The potential interference is between SBS2 (mainly C>T) and SBS7a, SBS7b (mainly C>T). This dataset presents synthetic tumors containing these three signatures in a simplified fashion, excluding the presence of any additional mutational signature. Scenarios 11, 12, E-11, and E-12 A set of 30 random mutational signature profiles based on SBS-96 classification and a set of 30 random 1,697 feature signature profiles (mimicking SignatureAnalyzer's composite signatures). Each of these sets of random signatures were used in two types of exposures, one with more (mean $15. 6) signatures per tumor and one with fewer (mean $4) signatures per tumor. Scenarios 13 and 14 A set of 2,700 synthetic whole-genome samples with mutational spectra matching the ones observed in PCAWG, including 300 spectra from each of 9 different cancer types.