dataesr/scientific-paragraphs-categorization huggingface dataset:
Context. Defining templates of galaxy spectra is useful to quickly characterise new observations and organise databases from surveys. These templates are usually built from a pre-defined classification based on other criteria. Aims. We present an unsupervised classification of 702248 spectra of galaxies and quasars with redshifts smaller than 0. 25 that were retrieved from the Sloan Digital Sky Survey (SDSS) database, release 7. Methods. The spectra were first corrected for redshift, then wavelet-filtered to reduce the noise, and finally binned to obtain about 1437 wavelengths per spectrum. The unsupervised clustering algorithm Fisher-EM, relying on a discriminative latent mixture model, was applied on these corrected spectra. The full set and several subsets of 100000 and 300000 spectra were analysed.
Introduction There is a long recognised need for templates of typical galaxy spectra to quickly characterise new objects. For instance, such templates are used in automatic classification in databases such as the Sloan Digital Sky Survey (SDSS), in which a χ 2 minimisation is used to separate galaxies from quasars (QSOs) and stars, followed by a categorisation into starbursts, star-forming galaxies and active galactic nuclei (AGN) using line ratios 1. Templates can be built by stacking spectra to reduce the noise that can make some features disappear  (Dobos et al. 2012). This strategy obviously requires perfectly identical spectra that can be obtained by visual inspection, by a classification made from some general or specific properties of the galaxies (e. g. line ratios, continuum, morphology, and model fitting of stellar population), or by machine learning tools.
It is a subspace clustering algorithm, based on a latent mixture model that fits the data into a low-dimensional discriminative subspace. Fisher-EM uses a modified version of the expectationmaximisation (EM) algorithm for model inference and introduces a Fisher criterion to optimise the quality of the clustering. It has been used only once in astrophysics up to now, on a data set of 100000 galaxies described by 12 parameters  (Siudek et al. 2018 ). In the present paper, we apply Fisher-EM on a larger sample of 702248 spectra (with 1437 wavelengths) from the SDSS. The paper is organised as follows. The data are described in Sect. 2. The Fisher-EM algorithm and its application on our large data set are detailed in Sect. 3. The results are given in Sect. 4 and an estimation of the statistical robustness of our classification is presented in Sect.
The data In this work, we use the same data set as studied by Sánchez Almeida et al. (  2010 ) and  De et al. (2016). Thus, we retrieved the spectra of 702248 galaxies and QSOs (with redshift smaller than 0. 25) from the SDSS database, release 7 2. For consistency with the two previous studies, we chose not to correct the spectra for galactic extinction. Apart from a philosophical justification explained in Sánchez Almeida et al. (  2010 ), the correction would be small (median extinction g-r of 0. 03 for our sample) and unavoidably somewhat uncertain, bringing an additional variance (see a further comment in Sect. 7. 1). The raw spectra have 3850 points in the wavelength range imposed by the instrument, λ = 3800 to 9250 Å. The spacing is uniform in resolution (δλ/λ = 1/4342). Taking the redshift into account, the range common to all the spectra goes from 3806 to 7371 Å with 2874 wavelengths.
The whole model is referred to by DLM [Σ k β k ] in  Bouveyron & Brunet (2012a). Several other models can be obtained from the DLM [Σ k β k ] model by relaxing or adding constraints on model parameters. For instance, the covariance matrices Σ 1. Σ K and the noise parameters β 1. β K can be assumed to be common to all classes (hence independent on k). This is the statistical model DLM  [Σβ]  (referred to as DB in the R package FisherEM). A list of the 12 different DLM models is given in Table  1  in  Bouveyron & Brunet (2012a). Such a family simultaneously yields very parsimonious models and allows us to carry out a fit in various situations. An estimation procedure, called the Fisher-EM algorithm, is also proposed in  Bouveyron & Brunet (2012a)  to estimate the discriminative space and parameters of the mixture model.
The choice of the best statistical DLM model and the optimum number of clusters depends on the data and are estimated with three likelihood criteria: the classical Akaike information criterion (AIC), the popular Bayesian information criterion (BIC) and the integrated completed likelihood (ICL). These criteria all penalise the likelihood by the number of parameters of the statistical model, but the BIC also includes the number of observations while the ICL in addition favours well separated statistical models  (Biernacki et al. 2000; Girard & Saracco 2016). Hence, the maximum value of ICL determines the best statistical model and the optimum number K of classes in the analyses performed in the present study. The Fisher-EM algorithm is implemented in the eponym package for R. For the present study, we used the version 1.
We are aware that this approach results in a somewhat complicated scheme, but in this exploring stage of the application of Fisher-EM on such a large astronomical sample, we believe it is essential to perform thorough and robust analyses to be confident with the resulting classification and to suggest tractable strategies to tackle even larger samples. These subsets were obtained simply by arbitrarily splitting the full sample assuming that the order within the SDSS database would provide enough randomisation of the galaxy diversity. We summarise our approach in Fig. 1  as follows: i) The full sample with 702248 spectra was only partially analysed, because the very good agreement between all our analyses did not seem to justify the large computation time that it would have required (Sect. 6).
The (Euclidean) distance of the test spectrum with all the spectra of the training sample is computed and the class is determined from the majority vote of the k closest spectra. Obviously this method takes much more time than the previous approach. In addition, the Fisher-EM classification is obtained in a subspace of the features (wavelengths), while kNN uses all features to compute the distance between spectra. However, the kNN algorithm is useful when there is no statistical model for the true classification. If the true classes are available only through their mean spectra, then k=1. When the class attribute is given for the whole training sample, then k=5 is a standard choice and is adopted in this study. The error rate provided in the R package mclust is used in this paper as an indicator of the quality of the match between two classifications.
Distinct sets of 100000 spectra The optimum number of classes in the subsets of 100000 spectra is between 21 to 61 (Fig. 1 ), hence always lower than 86. This may indicate that the diversity of spectra in the training sample is not represented in all the 100000 subsets. To ensure a better match between the classifications, we first need to reduce (merge) the reference 86 classes to the number of classes for each subset. This is done with the function mapClass from the package mclust in Ror by matching the mean spectra of the classes with kNN (k=1). The error rate is found to be between 9% and 21% with a mean at 15% for the six subsets of 100000 spectra and the subset with 102248 spectra.
Classifying complex spectra is certainly difficult, as illustrated for instance in  Rahmani et al. (2018)  in which a semi-supervised algorithm (self-organizing maps) is used to classify spectra against templates with a very small sample. However, we identify two possible caveats related to the use of templates. Firstly averaging spectra may lead to a loss of some precise and crucial information in the lines, thus we think this point should be addressed specifically on large samples. We intend to pursue the present work in this direction. In particular, we are completing two studies on simulated spectra of galaxies, one with simple single stellar populations (SSPs) and one using the CIGALE code  (Boquien et al. 2019). Secondly, we believe it is important to take the intra-class variance into account when comparing with templates or other classifications.
Acknowledgements. We thank the referee for constructive comments that significantly improved the paper. This work has been partially supported by the  French government , through the  3IA Côte d'Azur Investment in the Future  project managed by the  National Research Agency (ANR)  with the reference number  ANR-19-P3IA-0002. Funding for the  Sloan Digital Sky Survey (SDSS)  has been provided by the  Alfred P. Sloan Foundation , the Participating Institutions, the  National Aeronautics and Space Administration , the  National Science Foundation , the  U. S. Department of Energy , the  Japanese Monbukagakusho , and the  Max Planck Society. The SDSS Web site is http://www. sdss. org/. The SDSS is managed by the  Astrophysical Research Consortium (ARC)  for the Participating Institutions. The Participating Institutions are The  University of Chicago ,  Fermilab , the  Institute for Advanced Study , the  Japan Participation Group , The  Johns Hopkins University ,  Los Alamos National Laboratory , the  Max-Planck-Institute for Astronomy (MPIA) , the  Max-Planck-Institute for Astrophysics (MPA) ,  New Mexico State University ,  University of Pittsburgh ,  Princeton University , the  United States Naval Observatory , and the  University of Washington.
This is notably the case for the efficient scanning of large and complex microscopy samples (eg, Refs  [2] [3] [4] [5] [6]  ). For other important applications, such as real-time control of cellular processes (eg, Refs  [7] [8] [9] [10] [11] [12] [13]  ), results are generally obtained using ad hoc software solutions. Very few generic tools have been developed so far to facilitate the realization of complex, reactive microscopy experiments. One notable exception is Pycro-Manager  14. This powerful framework is built on top of µManager, a widely-used software  15, 16  controlling a large range of microscopy hardware. In Pycro-Manager, reactive protocols are built from the ground up. Whereas this gives maximal flexibility, this also increases the difficulty to rapidly design experiments, especially for non-expert users.
MicroMator Software In this paper, we present MicroMator, a simple software solution supporting reactive microscopy experiments, and demonstrate its potential via two challenging case studies. In MicroMator, events play a fundamental role (Fig  1A ). They consist of Triggers and Effects. They can be defined by the user in a flexible manner. Examples of triggers include "at the 10th frame", "if more than 100 cells are in the field of view", "if the fluorescence of the 3rd newborn cell exceeds 100 a. u. " and "if message !update position=10 frame=last is received from Discord". Examples of effects include changing a microscope configuration, sending light in the field of view with a given pattern, actuating a microfluidic pump, or starting an optimization routine. Microscopy experiments are defined by a main image acquisition loop, that serves as a backbone for the experiment, and by event creation functions (Fig  1B ).
Naturally, the main acquisition loop itself can be modified by event effects in the course of the experiment. MicroMator is written in Python 3, is open-source and has a modular design. For controlling hardware, MicroMator primarily uses the powerful Python API of µManager pymmcore, but can also use other dedicated Python or web-based APIs provided by vendors, as done for our CellAsic ONIX microfluidic platform. Various types of analysis can be performed using dedicated software modules, such as on-line image analysis or real-time control and optimization. Communication modules can be used to interface MicroMator with digital distribution platforms such as Discord to track experiment progress and potential issues. Lastly, MicroMator leverages Python's multiprocessing module to perform computations concurrently and possesses an extensive and customizable logging system, gathering logs of all modules in a unique file and fostering reproducible research (see SI Text).
We also provide SegMator, a software which uses U-Net for bright-field yeast cell segmentation (provided by DeLTA  18  ), and tracking using TrackPy  19. U-Net is a convolutional neural network with a structure that excels at image segmentation  20   To showcase the full potential of reactive experiments performed with MicroMator, we designed experiments in which cellular processes are controlled in real-time. Single-cell stimulations are computed on line based on the cell state and/or position, demonstrating that reactive loops can be implemented at the level of individual cells. These experiments are inspired by previously-published studies  10, 11, 13, 21  and show how these could be repeated and further extended using generic software. We also provide a tutorial application in which cells with fluorescent proteins are imaged with increasing durations such that the measured intensity reaches a given threshold (Supplementary Text 1).
Discussion We presented MicroMator together with two challenging applications demonstrating how this software can help using automated microscopy platforms to their full potential. Each application goes beyond the state of the art. We showed the first demonstration of control of protein expression at the single cell level in dense field of cells. This requires to jointly solve two challenges, namely obtaining sufficiently precise single-cell stimulations with DMDs and segmenting and tracking cells 7 with sufficient accuracy over extended durations. We also provide the first demonstration of cell recombination targeted at the single-cell level, enabling single-cell resolution patterns. In comparison with Pycro-Manager, MicroMator uses the Micro-Manager GUI to create a main acquisition backbone for the experiment and reactive events are then used to enhance or even dynamically modify this initial plan.
Events are created by default as separated threads and an extended logging system gathers messages from all modules that might be running in parallel. This structure provides robustness to real-time issues and facilitates error identification, two critical aspects for developing long and complex experiments. Yet, we foresee that reactiveness in microscopy will primarily be used to enhance and automate classical experiments. Examples of simple use cases abound: triggering autofocus only when needed, dynamically adjusting the imaging condition to the signal strength, identifying novel regions of interest, or following the course of experiments via easily accessible online services (e. g. warning messages sent on Discord), to provide but a few examples. Thanks to its modular nature and to its use of a simple but powerful event system, MicroMator capacities can be conveniently expanded to drive novel hardware or perform a wide range of analyses.
The current list of modules includes a Microscope Controller module, an Image Analysis module, a Model Predictive Control module, and a Discord Bot module. The Microscope Controller module is an interface with the Python wrapper for MicroManager pymmcore. The Image Analysis module uses deep learning methods to segment yeast cells from bright-field images. It also uses an efficient algorithm for cell tracking. This module is also available as a standalone tool called SegMator. The Model Predictive Control module implements state estimation and model predictive control routines for deterministic and stochastic systems, at either the population or single-cell level. The Discord Bot module uses a web app running on the microscope's computer and connected to the Discord communication system. MicroMator, SegMator, event definitions for representative experiments (Fig 2E and S4), and data analysis code for the experiments (Fig 2E, 3D, and S4), as well as a tutorial example (Supplementary Text 1), can be found online: https://gitlab.
inria. fr/InBio/Public/micromator. Raw and processed data for Fig 2C-E, 3C-D, and S4 are freely available on the zenodo repository: https://doi. org/10. 5281/zenodo. 4616659 (45GB). Supplementary movies  Movie S1: real-time_segmentation_and_tracking_with_SegMator. mov. Time-lapse movie showing the realtime segmentation and tracking quality obtained with SegMator. Left: bright-field image. Right: bright-field image overlaid with segmentation mask in cyan.  Movie S2: optogenetic_control_of_gene_expression-Open_loop. mov. Time-lapse movie showing the response of the cells (mScarletI fluorescence) in an open-loop control experiment. Corresponds to Fig. 2C.  Movie S3: optogenetic_control_of_gene_expression-Population_closed_loop. mov. Time-lapse movie showing the response of the cells (mScarletI fluorescence) in a population closed-loop control experiment.
Acknowledgments This work was supported by  ANR  grants  CyberCircuits  ( ANR-18-CE91-0002 ),  MEMIP  ( ANR-16-CE33-0018 ), and  Cogex  ( ANR-16-CE12-0025 ), by the  H2020 Fet-Open COSY-BIO  grant (grant agreement no. 766840 ) and by the  Inria IPL  grant  COSY. We acknowledge the support of the  U. S. Department of Energy  through the  LANL/LDRD Program  and the  Center for Non Linear Studies  for this work.
We analyzed bulk RNA-Sequencing (RNA-Seq) data from 535 patients (397 Ta, 135 T1, 3 CIS; an expansion of the 460 NMIBC patient cohort previously analyzed  8  ). Using unsupervised consensus clustering of gene-based expression values restricted to the 4000 genes with highest variation across the dataset  23  , we identified four transcriptomic classes which partially overlapped with the previous UROMOL2016 classes 1-3: high-risk class 2 was further subdivided into two subclasses, named class 2a and 2b for continuity (Fig. 1a, b ). Classes showed significantly different progressionfree survival (PFS, p = 6. 6 × 10 -5 ; log-rank test; Fig. 1c ): patients with class 2a tumors had the worst outcome, followed by patients with class 2b tumors. Patients with class 1 tumors had the best recurrence-free survival (RFS, p = 0.
Logistic regression models including continuous variables (EORTC, genome altered, and 12-gene progression score), EAU risk scores and T1HG subtypes showed no increased predictive value (Supplementary Fig. 8a-d ). Overall, each transcriptomic class has distinct clinical features, molecular characteristics, and tumor microenvironments, as summarized in Fig. 5c. To facilitate the use of the four transcriptomic classes in future research and clinical settings, we constructed a single-sample classifier for NMIBC. The classifier was built similarly to the recently published tool for the consensus subtypes of MIBC  27  , where a class label is assigned to the transcriptomic profile of a tumor based on correlation to the class-specific mean expression profiles (for details, see "Methods"). We applied the classifier to 14 independent cohorts, including three unpublished datasets, with a total of 1228 patients whose tumors were analyzed with a wide range of platforms (Fig.
The frequent FGFR3 mutations observed in class 1 and 3 suggest that FGFR inhibitors could be effective in these tumors, especially since the oral FGFR inhibitor BGJ398 recently showed antitumor activity in a marker lesion study of patients with NMIBC (ref. 63 , NCT02657486), and Pemigatinib (FGFR1,2,3 inhibitor) is being tested in an ongoing phase II clinical trial in patients with recurrent low-or intermediate-risk tumors (NCT03914794). Intravesical chemotherapy should be considered especially for class 3 tumors, but possibly also for class 1 tumors although the recurrence rate is lower in these patients. BCG response mechanisms have been studied intensely  64  and so far, one of the most promising markers of BCG response is fluorescence in situ hybridization (FISH, Urovysion) analysis of chromosomal abnormalities  65.
The BCG cohort was used to investigate time to BCG failure using multiple features available from our datasets. BCG failure-free survival was defined as time to first high grade tumor or first progression to MIBC after BCG treatment. DNA and RNA extraction. Tumor tissue was collected fresh from resection in each clinical center, embedded in Tissue-Tek O. C. T. and snap frozen in liquid nitrogen before storage at -80 °C. Total RNA was extracted from serial cryosections using RNeasy Mini Kit (Qiagen) and quantified using an Infinite 200 PRO NanoQuant spectrophotometer (Tecan). RNA integrity was assessed using a 2100 Bioanalyzer (Agilent Technologies) and only samples with an RNA Integrity Number (RIN) above five were included. DNA was extracted using Puregene DNA Isolation kit (Fischer Scientific).
Genes were ranked based on median absolute deviation (MAD) across all samples and divided into subsets of the top -2000, -4000, -6000, -8000, -10,000, -12,000 MAD-ranked genes. Consensus clustering was performed on the different gene subsets using the R package Con-sensusClusterPlus (settings: maxK = 10, reps = 1000, pItem = 0. 95, pFeature = 1, clusterAlg = "hc", distance = "pearson"). To identify the most representative samples within each cluster, silhouette scores were computed for all samples using the R package CancerSubtypes. A four-cluster solution based on the top-4000 MAD-ranked genes was chosen. Consensus clustering was furthermore performed on Ta low grade tumors only (n = 286) and T1 high grade tumors only (n = 101) to identify subtypes within pathologically homogeneous tumors.
Indels were not considered here due to technical issues that may arise from calling this from RNA-Seq data. Briefly, STAR v2. 7 was used to align the raw RNA reads to the hg38 human genome assembly and PICARD tools were used to mark duplicates. GATK tools, SplitNCigarReads, BaseRecalibrator, and ApplyBQSR were applied in order to reformat some of the alignments that span introns and correct the base quality score. Finally, the HaplotypeCaller software was used to call variants. The resulting VCF files were annotated using SnpEff followed by filtration for possible impact on proteins. First, only SNVs annotated with a HIGH or MODERATE impact by SnpEff were included and SNVs in splice-site genomic locations were excluded. Second, mutations with an rs ID in dbSNP were excluded. Third, only mutations with a quality score above 100 and a Fisher Strand score (FS) below 30.
RNA-based mutational signature analysis. To infer mutational signatures, we included mutations called within the gene sequence (HIGH, MODERATE, and LOW impact) and excluded mutations with rs ID together with mutations with a quality score below 100 or a Fisher Strand score (FS) above 30. 0. Finally, mutations were included if they met the following criteria: (1) alternate allele frequency (AF) > 0. 15 and < 0. 60; (2) number of reads > 20. Only samples with more than 100 SNVs were kept to infer the mutational signatures (n = 441). We used nonnegative matrix factorization to decompose the motifs matrix into seven signatures and their corresponding weights using the R package SomaticSignatures. The similarity between the seven inferred signatures and defined COSMIC signatures was examined using the R package MutationalPatterns.
Copy number analysis. GSA Illumina SNP arrays (~760 k positions) were used on tumor DNA from 473 patients in order to assess copy number alterations. We previously applied the Infinium OncoArray-500K BeadChipGenotyping arrays for the paired germline samples and used this as reference. LogR Ratio (LRR) and Ballele-fraction (BAF) were corrected and normalized using the Genotyping module from GenomeStudio 2. 0 (Illumina) within each array type and all positions uniquely found in both arrays were exported for further analysis (151,291 probes). The R package ASCAT was used for segmentation of the genome and we used the raw-segmented total copy number, the raw-segmented BAF data and various empiric thresholds (gains: > 0. 08, high gains: > 0. 16, loss: < -0. 1, high loss: < -0. 2, allelic imbalance (AI): < 0.