dataesr/scientific-paragraphs-categorization huggingface dataset:
Trace Matching -Input: dome. fits ; -Output: TraceMatch. fits To identify each individual spaxel trace, we run the "extract" method of sep 2 (  Barbary et al. 2016 ) -the Python implementation of Sextractor  (Bertin & Arnouts 1996)  -on a dome CCD-exposure. The sep function defines an ellipse contour on dispersed dome light for each spaxel; see Fig. 2. In the pysedm implementation for the SEDM, spaxel traces are defined as the rectangles formed by taking a 2. 4 pixels region above and below the spaxel ellipse centroid -so 5. 8 pixels in total -and starting 70 x-pixels to the left of the ellipse centroid and ending 230 xpixels to the right ; the dome light times the instrument response been mostly red. The 2. 4 pixel width has been selected to encapsulate 95. 5% (±2σ) of the spaxel light given the SEDM cross dispersion.
2004 ). We invite the reader to use the pyifu python library to open these cubes. A cube fits file contains five entries: The primary image contains an M × N array, where M is the number of spaxels and N is the length of each spectrum. The corresponding wavelength array can be reconstructed from the header, using the starting wavelength and the wavelength step recorded in the primary header as CDELT1 and CRVAL1, respectively. The second image, "VARIANCE", contains an M × N data array corresponding to the spaxel spectral variance. The third image, "MAPPING", has a 2 × M data array. This records the {x, y} coordinates of the M spaxel centers. The fourth image, "SPAX_ID", has an M-length array where the spaxel "id" is stored. This "id" corresponds to the "trace-id" and enables the user to easily map a given spaxel from a 3D-cube to its corresponding dispersion trace on the CCD 2D-image.
Spectral Extraction from 3D Cube Within two minutes of the end of a science exposure, a 3D cube fits file is created following the pysedm cube extraction detailed in section 2. Each spaxel of the 3D data cube includes three intensity components, each one characterized by its own spatial signature: (1) the target flux, from a point source located close to the center of the MLA, (2) the night sky flux, a spatially flat component over the entire IFU field of view, and (3) the (potential) host galaxy flux, a structured background. The core purpose of pysedm is to automatically extract the point source spectrum to enable the typing of the transient with no human interaction. However, for full flexibility and reliability, the code allows human interaction for potential fine-tuned re-extractions of complex cases (see section 3.
Some statistics for 19 weeks of operations Since the beginning of August 2018, pysedm has been the sole pipeline for the SEDM. The ZTF collaboration is using this instrument to type the transients discovered by the ZTF camera mounted on the Palomar 48" Schmidt (P48) telescope. We present in fig. 15  the fraction of object types generated by pysedm as a function of time, and the fraction of time spent acquiring SEDM observations. We spend, on average, 6h30m on targets each night. In addition, we have a typical overhead of 180s, including slew time and readout. We point out two dips in the observing time; the deepest in mid-October and a smaller dip in early December. These correspond to an engineering period (mid-October), and a period of poor weather conditions, notably proofs: manuscript no.
Acknowledgements. We thank  D. O. Cook ,  K. De ,  K. B. Burdge  and  A. Y. Ho  for generously sharing their spectra for the "SEDM and other spectrograph" figure. This project has received funding from the  European Research Council (ERC)  under the European Union's  Horizon 2020 research and innovation program  (grant agreement n • 759194 -USNAC). The SED Machine is based upon work supported by the  National Science Foundation  under Grant No. 1106171. Based on observations obtained with the  Samuel Oschin Telescope 48-inch  and the 60-inch Telescope at the  Palomar Observatory  as part of the  Zwicky Transient Facility  project. ZTF is supported by the  National Science Foundation  under Grant No. AST-1440341  and a collaboration including  Caltech , IPAC, the  Weizmann Institute for Science , the  Oskar Klein Center at Stockholm University , the  University of Maryland , the  University of Washington ,  Deutsches Elektronen-Synchrotron and Humboldt University, Los Alamos National Laboratories , the  TANGO Consortium of Taiwan , the  University of Wisconsin at Milwaukee , and  Lawrence Berkeley National Laboratories.
The main novelty of our work is to propose a model and its estimation at the master level by exploiting the sufficient statistics from the workers, in a DPM compliant approach. Our solution takes advantage of the computing power of distributed systems by using parallel frameworks such as MapReduce or Spark  [27]. Our DC-DPM solution distributes the Dirichlet Process by identifying local clusters on the workers and synchronizing these clusters on the master. These clusters are then communicated as a basis among workers for local clustering consistency. We modified the Dirichlet Process to consider this basis in each worker. By iterating this process we seek global consistency of DPM in a distributed environment. Our experiments, using real and synthetic datasets, illustrate both the high efficiency and linear scalability of our approach.
Massive Distribution and Spark Clustering via Dirichlet Process Mixture based on Gibbs Sampling is unable to scale to large datasets due to its high computational costs associated with Bayesian inference. For this reason, we aim to implement a parallel algorithm for DPM clustering in a massively distributed environment called Spark which is a parallel programming framework aiming to efficiently process large datasets. This programming model can perform analytics with in-memory techniques to overcome disk bottlenecks. Similar to MapReduce  [4] , Spark can be deployed on the Hadoop Distributed File System (HDFS)  [23]. Unlike traditional in-memory systems, the main feature of Spark is its distributed memory abstraction, called resilient distributed datasets (RDD), that is an efficient and fault-tolerant abstraction for distributing data in a cluster.
,K α j w u N j -1+α j ∫ F (y n,j ,ϕ)dG 0 (ϕ) new Update of α j Draw ϕ c for new clusters where the weight w c is the proportion of observations from cluster c evaluated on the whole dataset and w u the proportion of non affected observations (awaiting the creation, innovation, discover of their real clusters). Therefore, these parameters are updated at the master level during the synchronization. Now, the scale parameter α j can be viewed as a tuning parameter between local (worker) and global (master) proportions. Following  [9]  we use an inverse gamma prior to infer this parameter. This modification of the update implies a slightly modified DPM in each worker j : y n,j ∼ F (θ n,j ) θ n,j ∼ G j G j ∼ DP(α j ,G) α j ∼ IG(a,b) G = K c=1 w c δ ϕ c +w u G 0 , with w u + K c=1 w c = 1.
So, we suggest an update by clusters. In this view, we denote z j,k the master label of the cluster k in worker j. To take into account the worker information ({ϕ P(z j,k =c | z j,k ) ∝ #(c) N -1+γ F ( ȳj,k ,ϕ c ),c = 1,. ,K γ N -1+γ ∫ F ( ȳj,k ,ϕ)dG(ϕ | ϕ workerj k ) The labels {c n,j } of all the observations in the cluster k of worker j are then assigned to the master label z j,k. Next, the cluster parameters ({ϕ c } c=1,. ,K ) are updated from the posterior computed on the whole dataset. We assume that we don't need all the data but only sufficient statistics from all clusters from all workers to compute the posterior. This assumption is straightforward for many distributions, as the exponential family  [1]. Last, the synchronization of the workers is done through the definition of G using the updated parameters ({ϕ c } c=1,.
,K ) and with weights drawn from a Dirichlet distribution Dir (n 1 ,. ,n K ,γ ). The end user parameters of this Dirichlet distribution are updated at the master level from the whole dataset. The size n k is the sum of all observations having label k at the end of the master Gibbs sampling. (w 1 ,. ,w K ,w u ) ∼ Dir (n 1 ,. ,n K ,γ ) γ ∼ IG(c,d) By doing so, we do not have to consider label switching. Clusters are explicitly defined at the master level and parameter values are not updated in the worker. At the worker level, only innovation (creation of new clusters) is implemented. This is summarized by Algorithm 2. Algorithm 2 DPM at master level for each (j,k) do Draw z j,k from P(z j,k =c | {c} j,k , ȳj,k ,{ϕ},γ ) ∝ #(c) N -1+γ F ( ȳj,k ,ϕ c ),c = 1,. ,K γ N -1+γ ∫ F ( ȳj,k ,ϕ)dG(ϕ | ϕ workerj k ) Update of ϕ and (w 1 ,.
,w K ,w u ) The workflow of our DC-DPM approach is illustrated by Figure  2. It consists in 4 steps: (1) Identify local new clusters in the workers (2) Compute and send sufficient statistics and cluster sizes from each worker to the master (3) Synchronize and estimate cluster labels from sufficient statistics (4) Send updated cluster parameters and cluster sizes from master to workers Our first proposition concerns the synchronization and estimation of the DPM. It is done with a Gibbs sampling conditionally on the sufficient statistics instead of the whole dataset/individual observations. Our second proposition is a construction of a shared prior distribution updated at the master level and send to the workers' DPM. This distribution reflects the information/results collected from all workers and synchronized at the master.
EXPERIMENTS The parallel experimental evaluation was conducted on a cluster of 32 machines, each operated by Linux, with 64 Gigabytes of main memory, Intel Xeon CPU with 8 cores and 250 Gigabytes hard disk. The centralized approach is an implementation DPM in Scala, and was executed on a single machine with the same characteristics. The distributed algorithm we proposed is an approximation of a classic DPM, we will compare its properties to a centralized DPM implementation, on synthetic data and also in our use-case for digital agronomy. The first step of our process is a distributed K-means that sets the initial state (usually we set K to be one tenth of the dataset size). Reproducibility : All our experiments are fully reproducible. We make our code and data available at https://github. com/khadidjaM/ DC-DPM.
In the rest of this section, we describe the datasets in Section 5. 1 and our evaluation criteria in Section 5. 2. Then, in Section 5. 3, we measure the performances, in response time, of our approach compared to the centralized approach and also by reporting its scalability and speed-up. We evaluate the clusters obtained by DC-DPM in the case of real and synthetic dataset in Section 5. 4 and Section 5. 5 discusses the results and interest of our work in a real use-case of agronomy.
Datasets We carried out our experiments on a real world and a synthetic dataset. Our synthetic data are generated using a two-steps principle. In the first step we generate cluster centers according to a multivariate normal distribution with the same variance σ 2 1 for all dimensions. In the second step, we generate the data corresponding to each center, by using a multivariate normal distribution parameterized on the center with the same variance σ 2 2 for all dimensions. We generated a first batch of 5 datasets having size 20K, 40, 60, 80K and 100K with σ 2 1 = 1000 and σ 2 2 = 1. They represent 10 clusters. We generated a second batch of 5 datasets having size 2M, 4M, 6M, 8M and 10M with σ 2 1 = 100000 and σ 2 2 = 10. They represent 100 clusters. This type of generator is widely used in statistics, where methods are evaluated first on synthetic data before being applied on real data.
1. The Adjusted Rand Index (ARI): it is the corrected-for-chance version of the Rand Index  [24] , which is a function that measures the similarity between two data clustering results, for example between the ground truth class assignments (if known) and the clustering algorithm assignments. ARI values are in the range [-1,1] with a best value of 1. 2. The residual sum of squares (RSS): it is a measure of how well the centroids (means) represent the members of their clusters. It is the squared distance of each data from its centroid summed over all vectors. In the univariate case, the RSS value divided by the number of observations gives the value of the Mean Squared Error (MSE), an estimator of the residual variance. In multivariate dataset with independent variables, the RSS value divided by the number of observations gives an estimator of the sum of the variable variances.
Response Time In this section we measure the clustering time in DC-DPM and compare it to the centralized approach. Figure  3  reports the response times of DC-DPM and the centralized approach on our synthetic data, limited to 100K data points. Actually, the centralized approach does not scale and would take several days for larger datasets. The distributed approach is run on a cluster of 8 nodes. The results reported by Figure  3  are in logarithmic scale. The clustering time increases with the number of data points for all approaches. This time is much lower in the case of DC-DPM, than the centralized approach. On 8 machines (64 cores) and for a dataset of 100K data points, DC-DPM performs the clustering in 24 seconds, while the centralized approach needs more than 7 hours on a single machine.
Clustering Evaluation In the following experiments, we evaluate the clustering performance of DC-DPM and compare it to the centralized approach. Table  1  reports 1) the ARI value computed between the clustering obtained and the ground truth, 2) the RSS value divided by the number of data N and variance (σ 2  2 ), and 3) the number of clusters, obtained with DC-DPM and the centralized approach on our synthetic data while increasing the dataset size. The DC-DPM is run on a cluster of 8 nodes. DC-DPM performs as well as the centralized approach, there is a small gap in RSS values which is negligible compared to the gained time. Table  2  reports an extended view on the ARI value, and the RSS value divided by the number of data N and by the variance (σ 2  2 ), and number of clusters obtained with DC-DPM, while increasing dataset size (up to 10 million data points).
DC-DPM is run on a cluster of 16 machines. The performance keeps showing the maximum possible accuracy, even with a large number of data points. Figure  7  gives a visual representation of our 4M data points synthetic dataset. Each cluster is assigned to a color. Our goal is to retrieve these clusters. Figure  8  represents the results obtained by our approach on the data of Figure  7 , with 16 nodes. Each cluster is assigned to a different color. shows the performance of our approach with almost perfect results where the discovered clusters are the same as the actual ones from the data. This is confirmed by Table  2 , line 2.
CONCLUSION We proposed DC-DPM, a novel and efficient parallel solution to perform clustering via DPM on millions of data points. We evaluated the performance of our solution over real world and synthetic datasets. The experimental results illustrate the excellent performance of DC-DPM (e. g. , a clustering time of less than 30 seconds for 100K data points, while the centralized algorithm needs several hours). The results also illustrate the high performance of our approach with results that are comparable to the ones of the centralized version. Overall, the experimental results show that by using our parallel techniques, the clustering of very large volumes of data can now be done in small execution times, which are impossible to achieve using the centralized DPM approach. A nice perspective of our approach is to open fundamental research tracks such as DPM clustering on complex data like, e.
1b ). For whole mouse brains (≈ 1 cm 3 ), typical datasets are isotropic (6. 5 µm sampling), small (12-16 GB), and they are recorded quickly (7-8 minutes) with minimal shadow artifacts. Together with standardized quick-exchange holders, these features allow for the fast screening of samples. With a travel range of 44×44×100 mm, very large samples such as a whole mouse central nervous system can be imaged. After acquiring overview datasets, users can zoom in and record multidimensional. CC-BY-ND 4. 0 International license a certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under The copyright holder for this preprint (which was not this version posted March 18, 2019. ; https://doi. org/10. 1101/577122 doi: bioRxiv preprint 3 data at higher resolution by mosaic acquisitions, for example revealing cellular distribution and long-range axonal projections of Purkinje cells in the mouse cerebellum (Fig.
1c-e ) or fine neurites in the developing nervous system of a chick embryo (Fig. 1f ). The instrument has been tested in combination with all common clearing methods ranging from CLARITY  5  , CUBIC  6  to organic solvent approaches such as iDISCO  7  and BABB 2. Due to the modular design of the mesoSPIM, switching between different imaging media can be done in less than a minute. The mesoSPIM setup, performance characterization, and example applications are presented with detail in Supplementary Notes including Supplementary Figures  1-30. Inspired by the openSPIM 8 and openSPIN  9  projects, documentation for the mesoSPIM hardware and software (written in Python) is freely available (https://github. com/mesoSPIM). Currently, there are 5 mesoSPIM setups in operation across Europe. The mesoSPIM is the ideal instrument to quickly bridge scales from the µm-to the cm-level, and serves as an excellent tool for detailed three-dimensional anatomical investigations in neuroscience and developmental biology.
Acknowledgements This work was supported by grants from the  Swiss National Science Foundation  ( 310030B_170269 , F. H. ;  31003A-153448 ,  31003A_173125 ,  CRSII3_154453 , and  NCCR  Synapsy  51NF40-158776 , A. H. ), the  European Research Council  ( ERC Advanced Grant BRAINCOMPATH , project  670757 ; F. H. ),  ERC   Starting Grant  (MULTICONNECT, project  639938 ; AR), the  Dutch science foundation (NWO VIDI Grant , project  14637 ; AR), and a gift from a private foundation with public interest through the  International Foundation for Research in Paraplegia  (A. H. and S. P. ) In addition, we would like to thank  Dubravka Göckeritz-Dujmovic  and  Sandrine Bichet  for help with sample preparation and  Martin Wieckhorst  for help with custom electronics.
Code availability The mesoSPIM software and documentation are available on Github (https://github. com/mesoSPIM). mesoSPIM-control is licensed under the GNU General Public License v3. 0 (GPL v3). CC-BY-ND 4. 0 International license a certified by peer review) is the author/funder, who has granted bioRxiv a license to display the preprint in perpetuity. It is made available under The copyright holder for this preprint (which was not this version posted March 18, 2019. ; https://doi. org/10. 1101/577122 doi: bioRxiv preprint.
Land subsidence in urban environments is an increasingly prominent aspect in the monitoring and maintenance of urban infrastructures. In this study we update the subsidence information over Rome and its surroundings (already the subject of past research with other sensors) for the first time using Copernicus Sentinel-1 data and open source tools. With this aim, we have developed a fully automatic processing chain for land deformation monitoring using the European Space Agency (ESA) SentiNel Application Platform (SNAP) and Stanford Method for Persistent Scatterers (StaMPS). We have applied this automatic processing chain to more than 160 Sentinel-1A images over ascending and descending orbits to depict primarily the Line-Of-Sight ground deformation rates. Results of both geometries were then combined to compute the actual vertical motion component, which resulted in more than 2 million point targets, over their common area.
Introduction Since the launch of Copernicus Sentinel-1A on 8 April 2014, a new era of continuous monitoring using spaceborne Synthetic Aperture Radar (SAR) sensors has started. Sentinel-1 constitutes a significant improvement from previous European C-band SAR missions, European Remote Sensing (ERS) satellites and Environmental Satellite (ENVISAT), since it reduced the temporal revisit time from 35 to six days, at best using the two satellite segments A and B, with a large swath coverage of 250 km. The scientific communities as well as Earth Observation (EO) practitioners were thus given the means to extend the use of spaceborne SAR data to land applications. In support to the EO community, the European Space Agency (ESA) continued developing appropriate tools for the utilization of the Copernicus Sentinel data.
By evolving existing tools, such as the Next ESA SAR Toolbox (NEST) as well as integrating others, the SeNtinel Application Platform (SNAP)  [1]  becomes a multi-mission toolbox supporting both SAR and optical data processing. Newly implemented on Sentinel-1, the Terrain Observation by Progressive Scans (TOPS) acquisition mode  [2]  required further development in terms of interferometric handling to ensure robust results. The SNAP TOPSAR) capabilities were made available to users at an early stage, just before the start of Sentinel-1 data dissemination, while SNAP TOPS Interferometric SAR (InSAR) development were first communicated at the ESA Fringe 2015 consultation meeting. Currently, TOPS InSAR processing is sufficiently documented  [3] [4] [5] [6] [7]  and SNAP remains a widely used end-to-end open source tool for processing of Sentinel-1 data.
Further development of SNAP was carried out to include exports to software packages supporting more advanced interferometric analysis, such as Stanford Method for Persistent Scatterers (StaMPS)  [8]. StaMPS is a freely distributed package for research purposes with a large user community that incorporates Persistent Scatterer Interferometry (PSI) and Small Baseline methods to measure ground displacements from time series of SAR acquisitions. The idea was for the open source InSAR processor to be used together with StaMPS PSI, boosting the utilization of the Copernicus Sentinel-1 data for geohazard-related applications. The potential of SNAP-StaMPS integration has been already demonstrated at  [9] , for which the authors had also published a set of scripts to support utilization by the scientific community  [10].
In this study we employ such open tools to analyze the urban deformation of the Rome metropolitan area for the first time using Sentinel-1 data, and combined ascending and descending orbits to depict the vertical urban deformation with special attention to different areas. Ground deformation analyses of Rome have already been undertaken in the past  [11, 12] , but these have not been updated recently, and have not included Sentinel-1 SAR observations. Here we intend to update existing knowledge with contemporary information regarding ground deformation in the Rome metropolitan area using open data and tools. Previous relevant studies are indirectly used to verify our findings and to allow us to understand which subsidence patterns correspond to already identified phenomena and which new sources of deformation that would require further attention.
Open Source Toolboxes This work has been carried out using the open source ESA SNAP and StaMPS software packages. The SNAP Graph Builder operator can be used to create processing chains which can be called using the batch mode Graph Processing Tool (GPT). We have exploited this utility to create the several templates necessary for creating single master TOPSAR coregistration and interferogram generation. Finally, in order to fully automate the single master interferogram generation, we have developed and made available, based on well-designed SNAP graphs, a set of scripts called "snap2stamps". These scripts enable automatic processing after setting some parameters in a configuration file. In fact, they are python wrappers which use the aforementioned templates based mainly on SNAP TOPSAR interferometric operators and whose outputs are compatible with StaMPS PSI chain.