dataesr/scientific-paragraphs-categorization huggingface dataset:
Our investigation is based on data from the COSMOS galaxy surv e y  (Scoville et al. 2007 ) , which provides excellent space-based imaging of galaxies o v er a wide range of redshifts and stellar masses, and is therefore ideal for our analysis. Ho we ver, the assumptions on which the reconstruction of the 3D shape distribution is based are highly simplistic which may induce uncertainties in our as well as in previous analyses. Our second goal is therefore to test several of these assumptions and to assess the o v erall performance of the reconstruction method. For that purpose we employ for the first time two state-of-the-art cosmological hydrodynamic simulations of galaxy formation, Horizon AGN and Illustris TNG. These simulations provide 3D as well as projected 2D galaxy morphologies which allows for an examination of the reconstruction method under controlled conditions.
Shapes We use galaxy shape estimates from the public Advanced Camera for Surveys General Catalog (ACS-GC, 3 Griffith et al. 2012 ). This catalogue is based on Hubble Space Telescope ( HST ) imaging in the optical red I AB broad-band filter F814W. The absence of atmospheric distortions allows for an excellent image resolution, which is mainly limited by the width of the HST point spread function (PSF) of 0. 085 in the F814W filter and the pixel scale of 0. 03. Sources were detected using the GALAPAGOS software  (H äußler et al. 2011 ) , which runs SEXTRACTOR  (Bertin & Arnouts 1996 )  and GALFIT  (Peng et al. 2002 )  in two subsequent steps. Galaxy shapes are described by the two-dimensional major o v er minor axis ratios q 2 D , which are derived by GALFIT from fits of a single S érsic model to each objects image.
Morphological classification For the selection of disc-dominated galaxies we use the Zurich Structure & Morphology Catalog 4  which is derived from the same HST imaging data as the ACS-GC. It provides a morphological classification for each galaxy, derived with the Zurich Estimator of Structural Type and is referred to as ZEST catalogue in the following. The ZEST classification is based on a principal component analysis of five non-parametric diagnostics: asymmetry, concentration, Gini coefficient, 2nd-order moment of the brightest 20 per cent of galaxy pixels (M 20 ) and ellipticity (see  Scarlata et al. 2006 Scarlata et al. , 2007 ; ; Sargent et al. 2007  , who also provide validations of their classification). The catalogue contains galaxies in the COSMOS field brighter than I AB = 24. We select objects, which i) are classified as galaxies ( [acs mu class, stellarity] = = [1,0] ), ii) do not reside in automatically or manually masked regions ( [acs mask, acs masked] = = [0,1] ) and iii) are not flagged as unusable or spurious ( [acs clean, junkflag] = [1,0] ).
Matched catalogue Matching objects in these three COSMOS catalogues is not straightforward due to variations in common properties such as positions and magnitudes, which can lead to spurious mismatches. These variations can originate from atmospheric distortions in the ground-based COSMOS2015 data, differences in the employed image analysis software (i. e. SEXTRACTOR and GALFIT ) and its configuration, in the employed telescopes, cameras and filters, as well as different quality cuts applied before matching. In order to minimize the chance for mismatches, galaxies are matched based on angular positions as well as on magnitudes. We start by matching objects in the COSMOS2015 and ACS-GC catalogue in three steps. 1) We select pairs of galaxies as candidate matches if their angular separation is < 0.
Horizon-AGN The hydrodynamic simulation HAGN was produced with the gridbased adaptive-mesh-refinement code RAMSES  (Teyssier 2002 ) , using cosmological parameters compatible with the constraints from WMAP-7  (Komatsu et al. 2011 ). The simulation includes the key processes rele v ant for galaxy formation: cooling, heating, and chemical enrichment of gas, the formation and evolution of stars and black holes as well as feedback from stellar winds, supernov ae and Acti ve Galactic Nuclei (AGNs)  (Dubois et al. 2014 ). Galaxies were identified in the distribution of stellar particles as groups with more than 50 members using the AdaptaHOP finder (Aubert,  Pichon & Colombi 2004 ). A key achievement of HAGN with respect to previous generations of cosmological hydrodynamic simulations is the broad diversity of galaxy morphologies, with realistic fractions of discy and elliptical objects  (Dubois et al.
Illustris TNG100 The TNG100  (Marinacci et al. 2018 ; Naiman et al. 2018 ; Nelson et al. 2018 ; Pillepich et al. 2018b ; Springel et al. 2018  ) is a magneto-hydrodynamic simulation, produced with the moving-mesh code AREPO  (Springel 2010 ) , which was run with cosmological parameters from the Planck Collaboration ( 2016 ). It includes the same key processes for modelling galaxy formation as HAGN, although with significantly different implementations which are described in  Pillepich et al. ( 2018a ). Galaxies are identified in dark matter subhaloes with non-zero stellar components. These subhaloes are detected in friends-of-friends groups by the SUBFIND algorithm  (Davis et al. 1985 ; Springel et al. 2001 ; Dolag et al. 2009 ). While it has been calibrated to match the galaxy mass function at z = 0, the simulation has been shown to predict main characteristics of observed galaxy populations reasonably well, such as morphological diversity (Rodriguez-Gomez et al.
(  2019 ). These images have been produced using the SKIRT radiative transfer code 7   (Baes et al. 2011 ; Camps & Baes 2015 )  by the Sloan Digital Sk y Surv e y in the g and i broad-band filters. The axis ratios are obtained from the second-order moments of the pixels in these images that belong to the source. Since these measurements are not PSFcorrected, we do not use them for testing the shape reconstruction method. Instead, we use them to obtain a rough estimation of how strongly the observed galaxy shapes depend on the wavelength range in which they are measured in order to interpret the COSMOS observations in Appendix E.
F or generating the predictions we choose a sample size of 10 5 , which is a compromise between a fast computation, needed for efficiently estimating the posterior, and having errors on the prediction which are negligible compared to those on the data vector. The latter condition is satisfied, as the number of galaxies in our six COSMOS subsamples is more than two magnitudes smaller than the samples used for generating the predictions (see Table  5  ). We estimate P ( θ | d ) by sampling the parameter space with the Markov chain-Monte Carlo MNRAS 515, 3603-3631 (2022) (MCMC) method using the code emcee 9 (F oreman-Macke y et al. 2013 ). For each posterior we run 32 independent chains with at last 1 000 steps each. The best fit parameters are obtained from the position of the maxima of the marginalized posterior distribution.
AC K N OW L E D G E M E N T S KH acknowledges support by the  Swiss National Science Foundation  (Grant No. 173716 ,  198674 ) and the  Forschungskredit grant  of the  University of Zurich  ( Projekt K-76106-01-01 ). CL acknowledges support from the  Programme National Cosmology and Galaxies (PNCG)  of  CNRS/INSU  with  INP  and  IN2P3. This work is part of the  Delta ITP consortium , a program of the  Netherlands Organisation for Scientific Research (NWO)  that is funded by the  Dutch Ministry of Education, Culture and Science (OCW). The research of JD is supported by the  Beecroft Trust and STFC. Our analysis is based on data products from observations made with  ESO Telescopes at the LaSilla Paranal Observatory  under  ESO  programme  ID 179. A-2005 and on data products produced by TERAPIX and the Cambridge Astronomy Surv e y Unit on behalf of the  UltraVISTA consortium.
Deep learning has revolutionized the automatic processing of images. While deep convolutional neural networks have demonstrated astonishing segmentation results for many biological objects acquired with microscopy, this technology's good performance relies on large training datasets. In this paper, we present a strategy to minimize the amount of time spent in manually annotating images for segmentation. It involves using an efficient and open source annotation tool, the artificial increase of the training dataset with data augmentation, the creation of an artificial dataset with a conditional generative adversarial network and the combination of semantic and instance segmentations. We evaluate the impact of each of these approaches for the segmentation of nuclei in 2D widefield images of human precancerous polyp biopsies in order to define an optimal strategy.
Introduction Over the last decade, deep learning approaches have outperformed all existing methods for image segmentation  [1] [2] [3] [4]. Semantic segmentation, the estimation of a label at each pixel, and instance segmentation, the identification of individual objects, were successfully applied to spatially characterize biological entities in microscopic images  [5] [6] [7] [8]. However, these powerful approaches rely on large annotated datasets. While more and more datasets become publicly available  9, 10  , annotated data for every combination of modalities, tissues and biological objects is far from completion. If the number of images to be segmented is low, a fully manual workflow might be the most time efficient option. Otherwise, procedures to efficiently build training datasets are required to use the full potential of deep learning-based segmentation at a single biological lab scale.
In this paper, we propose a strategy to minimize the amount of time dedicated to manually annotate images and investigate several approaches to maximize accuracy when only using one annotated image. We apply this strategy to segment nuclei stained with DAPI in widefield images of human colorectal adenomas (i. e. precancerous polyps) as follows. First, we take advantage of existing training datasets  11, 12  and massive data augmentation to obtain a preliminary segmentation. We then use an open source annotation software  12  to manually correct this segmentation and consequently define the training dataset. Next, we simulate synthetic images using a conditional generative adversarial network (GAN)  13  to increase the size of the training dataset. Finally, we combine U-Net  14, 15  , a semantic segmentation approach, and Mask R-CNN  16  , an instance segmentation approach, to improve the nuclear segmentation accuracy.
Image acquisition DAPI was used for nuclear counterstaining. Stained slides were mounted with ProLong™ Gold Antifade Reagent (Cat. # P36934, ThermoFisher) and imaged using the Akoya Vectra® Polaris™ Automated Imaging system (Akoya Biosciences, Marlborough, MA). Whole slide scans were done at 20X magnification and regions of interest where chosen randomly. Deep learning code U-Net, Mask R-CNN and pix2pix were coded in Python using the Python libraries numpy  17  , tensorflow  18  , keras  19  , scipy  20  and scikit-image  21.
Training dataset The training dataset consisted of three 1868 × 1400 images manually annotated with Annotater  12. Only one image was used to train U-Net and Mask R-CNN as well as pix2pix (conditional GAN) for most of the study, in addition to publicly available datasets (image set BBBC039v1 available from the Broad Bioimage Benchmark Collection 9 and a mouse intestinal epithelium dataset  12  ). The two other images were added to the training dataset in the last section to be compared with the combination of results obtained with U-Net and Mask R-CNN (see Figure  3a ).
U-Net training The annotated 1868 × 1400 image was divided into six 622 × 700 images for training: five of these images were included in the training dataset while the last one defined the validation dataset. As U-Net is a semantic segmentation approach, three classes were defined to allow separating nuclei as proposed in 22: inner nuclei, nuclei contours and background. To facilitate nuclei separation, the nuclei contours between touching cells were dilated  22. To limit over-fitting, the imaging field for images in the training dataset was set to 256 × 256 by randomly cropping the 622 × 700 input images. These cropped images were then normalized to obtain intensity values between 0 and 1. A root mean square prop was used to estimate the parameters of the deep convolutional neural network by minimizing a weighted cross entropy loss to handle class imbalance for 100 epochs without data augmentation and 25 epochs with data augmentation.
Amendments from Version 1 In this new version, we have changed the first section by: 1. Adding a comparison with a Stardist model trained on the 2018 Data Science Bowl, which is available on Fiji, 2. Better explaining the way we trained U-Net and Mask R-CNN to obtain the results shown in Figure  1. We also have toned down the benefit of using a conditional GAN to expand the size of the training dataset as it only improves marginally the segmentation accuracy. Finally, we have completely rewritten the discussion to present observations made in the manuscript rather than a universal guideline. Mainly: 1. The use of publicly available datasets and massive data augmentation are beneficial to build a training dataset and are now common practices in the field, 2. The conditional GAN approach does not improve drastically the segmentation accuracy, 3.
Combining instance and semantic segmentations lead to a substantial increase in segmentation accuracy and has the potential to be widely adopted in the field. Any further responses from the reviewers can be found at the end of the article modifications with the imgaug python library to simulated images with pix2pix. When combining the annotated image for this study with simulated images and/or existing datasets, the number of augmented images was defined to be balanced between the different data. U-Net post-processing An ImageJ macro  24, 25  was used to convert the three classes obtained with U-Net to individual nuclei. More specifically, individual nuclei were identified by thresholding the subtraction of the nuclei contours component from the inner nuclei component with a threshold equal to 0.
Mask R-CNN The annotated 1868 × 1400 image was divided into thirty-five 266 × 280 images for training: thirty of these images were included in the training dataset while the last five images defined the validation dataset. Version 2. 1 of Mask R-CNN  16  was used in this study. The backbone network was defined as the Resnet-101 deep convolutional neural network  28. We used the code in 5 to define the only class in this study, i. e. the nuclei. A data augmentation to increase the training dataset by a factor of 100 was processed before normalization with the imgaug python library  23  and included resizing, cropping, flipping, rotation, shearing, pixel dropout, blurring, sharpness and brightness modifications, noise addition and contrast modifications. Transfer learning with fine-tuning from a network trained on the coco dataset 29 was also applied.
In the first epoch, only the region proposal network, the classifier and mask heads were trained. The whole network was then trained for the next three epochs. In Figure  2  and Figure  3 , augmented simulated images were obtained by applying the same modifications with the imgaug python library to simulated images with pix2pix. When combining the annotated image for this study with simulated images and/or existing datasets, the number of augmented images was defined to be balanced between the different data. The maximum image size used for processing Mask R-CNN was larger than 256 as resizing and cropping were applied for data augmentation and set to 512. This parameter was defined as 1024 when other existing datasets were included for training as magnification in these images is higher.
Conditional GAN The annotated 1868 × 1400 image was divided into thirtyfive 256 × 256 images for training. As defined in 13, U-Net  14  was used for the generator and a convolutional PatchGAN classifier was used for the discriminator. Once trained, nuclei masks had to be generated to simulate images. Distributions for the number of nuclei per image and the size of nuclei were defined from the training dataset. The number of nuclei per image was then modeled as a Gaussian distribution while the size of nuclei was modeled by a Gumbel distribution to reflect the heavy tail distribution observed in the training dataset. Nuclei masks were then defined as ellipses randomly generated with these distributions with random orientation and a ratio between the two axes defined according to a Gaussian distribution of average s/π and standard deviation of 0.
Deep learning-based instance segmentation with existing datasets and massive data augmentation is used to initialize the training dataset A training dataset is required to train a deep learning method for object segmentation. While new approaches emerge for this task such as interactive machine learning  30  , users most often start with manually annotating objects of interest with existing annotation tools  31, 32. As shown in Figure  1a , this task is particularly challenging in our case due to the wide range of morphologies and high density of nuclei in polyps. We use the ImageJ plugin Annotater  12  to efficiently annotate nuclei, a task that takes approximately 30 hours for the image shown in Figure  1a. To avoid a fully manual annotation and save time, it is possible to use the same plugin to correct a nuclei segmentation obtained with an existing method.
Another possibility is to train deep learning approaches with existing training datasets. We propose to train a U-Net model and a Mask R-CNN model with a high throughput chemical screen on U2OS cells dataset (CC) (image set BBBC039v1 available from the Broad Bioimage Benchmark Collection  9  ) and a widefield mouse intestinal epithelium dataset (MIE)  12. These models are then used to segment the image shown in Figure  1a. While U-Net demonstrates a poor performance (Figure  1b ), Mask RCNN clearly surpasses the watershed approach and the pretrained Stardist model (Figure  1c ). When compared to the latter, the good performance of Mask R-CNN is explained by the fact that the MIE dataset includes epithelial nuclei, even though they come from mice. Correcting this segmentation with Annotater takes about 15-20 hours, which is clearly faster than an annotation from scratch.
Increasing the training dataset by using a conditional GAN improves nuclear segmentation accuracy When only considering the annotated image in Figure  1a  in the training dataset, U-Net leads to higher segmentation accuracy than Mask R-CNN (Figure  2a-b ). To increase the training dataset, we use the same annotated image to train a conditional Generative Adversarial Network (GAN)  13  and simulate images showing nuclei from masks defined as random ellipses generated with the distributions of nuclei size and nuclei number observed in the training dataset (see Figure  2c  and Methods). Only using simulated images lead to a lower accuracy for both deep learning approaches, even though applying mathematical operations to these synthetic images (augmented simulated training dataset, see Methods) improves the segmentation accuracy.
However, pooling together augmented simulated images and the annotated image from Figure  1a  slightly improves U-Net performance and distinctly increases the number of accurately identified nuclei with Mask R-CNN while decreasing the segmentation localization precision. Finally, adding existing datasets clearly leads to the optimal results for Mask R-CNN while degrading the accuracy for U-Net, which is consistent with the inability for this approach to generalize nuclear segmentation for different data as shown in Figure  1b. Overall, U-Net marginally benefits from using simulated images (red curve versus black curve in Figure  1a ) while the main gain for Mask R-CNN comes from the use of CC/MIE datasets and data augmentation (orange curve versus red and black curves in Figure  1b ).
Combining semantic and instance segmentations improves nuclear segmentation accuracy Nuclei segmented with Mask R-CNN show a higher localization precision than those obtained with U-Net as shown in Figure  1a-b. However, nuclei that are harder to delineate are missed with Mask R-CNN while U-Net accurately identifies pixels that belong to nuclei, even though the separation between individual nuclei might not be precise. In order to get the best of both worlds, we propose to combine the results obtained with U-Net trained with one annotated image with data augmentation and augmented simulated images, and the results obtained with Mask R-CNN trained with one annotated image with data augmentation, augmented simulated images and existing datasets with data augmentation (see Methods). As shown in Figure  3 , these results demonstrate a higher F1 score for any IoU threshold than obtained with U-Net or Mask R-CNN trained with 3 times more annotated images.
Discussion This study explores several strategies to minimize the amount of manually annotated data required to successfully train a deep learning model for instance segmentation. As already established in the field, the use of existing training datasets, even though modalities and/or tissues differ, allows to train instance segmentation models and obtain results on targeted data that can be manually corrected to initialize a new training dataset. The use of massive data augmentation is another well known approach to drastically increase the segmentation accuracy. While the use of conditional GANs to expand the size of the training dataset seems promising, the gain in accuracy shown in this study is not very convincing. The simulation pipeline used to generate the masks might have been too simplistic.
4. The points 1 and 2 are already well established in the field and will be systematically done nowadays, with almost any DL networks when data is available. Segmentation dataset are available as the authors highlight. So these aspects are sanity checks here and not novel implementations. However, it is reassuring to see that augmentation and use of pretrained models are helpful here as well. The more interesting aspects of this work lie in the use of GAN for expanding the size of the training dataset from an annotated image and the combination of output. Although the use of GAN makes sense for this application, the gains from such approach are clearly quite marginal as can be seen on Figure  2a  and 2b comparing the black and red lines, while the main gains are again from the use of additional dataset and augmentation as observed on Figure  1.
It's an important observation but maybe not as essential to the pipeline as is described in the manuscript as it stands. I suggest toning down the importance of this and clearly highlighting that the gains are in fact low here. Maybe the authors could further discuss why the gains are only small here: maybe the simulation pipeline from the masks to generate the training dataset of pix2pix is too simplistic for instance, wider range of shapes, background lights, heterogeneity of intensities or patterns on the nuclei etc. On the contrary it's quite clear that the combination of U-Net and MaskRCNN output are beneficial to the overall performance of the method and that's nicely shown here. I think that combining DL model outputs is currently underused and this is a nice additional demonstration of this here.
○ StarDist (https://github. com/stardist/stardist) from Uwe Schmidt and Martin Weigert, is an excellent tool for nuclei segmentation and is not included here. I suggest that the authors compare their IoU curves to those obtained from the pre-trained models provided by the method (even as Fiji plugin). This will give the readers a baseline on which to compare the approaches described here, which still require an investment in time to train multiple models and annotations.