dataesr/scientific-paragraphs-categorization huggingface dataset:
To improve accuracy, we tested a deep learning approach for nuclear marker identification. As the goal is not to identify regions but to make a decision for each nucleus regarding the presence/absence of an E2F or the diffuse/punctate/absence of EdU or pH3, the instance segmentation and U-Net approaches are not suitable. In contrast, with an input defined as an image patch centered on each nucleus, the Inception-V3 architecture is appropriate to decide about the presence, potentially in diffuse or punctate state, or absence of a marker. In addition to data augmentation, we also define a so-called pixel-based training dataset (as opposed to nuclei-based training dataset) that includes the input patches centered at each pixel belonging to the nuclei (see Materials and methods). This strategy has a similar effect to data augmentation, as it drastically increases the training dataset.
The latter might appear surprising, but the two different patterns for pH3 in confocal images are different enough to allow a strategy based on the thresholded area in the nuclei (see Materials and methods) to lead to satisfying results (Fig  2a ). However, the results for E2F4, Fig 2b, are not as good: Because E2F4 is also cytoplasmic, the extra-nuclear fluorescence confounds the thresholding decision. Finally, thresholding clearly fails to identify the two different patterns (diffuse, punctate) of EdU in images from both modalities as well as the patterns of pH3 in widefield images. In contrast, the Inception-V3 approach yields accuracies greater than 90% for all markers except E2F4 (89%) (Fig  2 ). As shown in S7 Table, the use of the pixel-based training datasets does not significantly improve marker identification for confocal images, but does improve performance on E2F3A, E2F4 and pH3 markers in widefield images (see S8 Table ).
Discussion This study demonstrates that the Mask R-CNN approach, coupled with transfer learning and data augmentation, can produce highly accurate nuclear segmentation of fluorescent images, even in a complex tissue such as the intestinal epithelium, by considering a small training dataset. Outstanding segmentation results are achieved for different imaging modalities, notably including widefield fluorescence microscopy, a ubiquitous imaging mode in cell biology research labs that produces challenging images of low sharpness and poor boundary definition. This is of major interest, as segmentation is a key task in image analysis of mammalian tissues where variability across tissue types is high and the number of modalities large and ever increasing. Therefore, the quest for comprehensive training datasets will be long and biology laboratories need efficient strategies to process their own images.
In this context, the Inception-V3 approach also obtains satisfactory segmentation accuracy, but the computation time, especially for training, represents a major limitation. Though the U-Net approach has demonstrated its utility in many biomedical studies, it is unable to generalize the ability to segment nuclei when the images are acquired with different modalities. Our results also attest to the superiority and efficiency of the deep learning approach for identifying the presence and state of nuclear markers. As the goal is to make a decision for each cell, the Inception-V3 architecture is well adapted to the task, computation time not being the issue that it is for nuclear segmentation. Considering pixel-based input patches improves performance if the training dataset is small. Moreover, transfer learning from nuclear segmentation improves marker identification.
This is explained by the fact that image features that are suited for nuclear segmentation are also meaningful for identifying nuclear markers. This is an interesting observation, as the same strategy could be used in an n-steps pipeline analysis. Indeed, a deep learning classifier could be trained for the first analysis applied to a particular dataset, the resulting parameters used to initialize the classifier for the second analysis, and so on. While deep learning approaches for both nuclear segmentation and nuclear marker identification show high accuracy, some level of discrepancy between desired and actual results is expected. Thus, it is important to allow the user to correct the results if higher accuracy is required. This need for interactive post-processing of the results motivated us to design and implement the ImageJ plugin Annotater.
Nuclei segmentation The five deep learning approaches were coded in Python with the Python libraries numpy  [42] , tensorflow  [43] , PyTorch  [44] , keras  [45] , scipy  [46]  and scikit-image  [47]. The code with the parameters used to train and process all experiments presented in this manuscript is available at https://github. com/tpecot/NucleiSegmentationAndMarkerIDentification/tree/master/. It was written with jupyter notebooks and widgets in order to be used by biologists. Video tutorials are also available on the same web page. Training dataset. The training dataset for nuclei segmentation consisted of twelve confocal and forty-five widefield 512 x 512 images annotated by a moderately-skilled researcher. Four different training datasets were used to evaluate the deep learning approaches and are summarized in S3 Table For the Inception-V3 approach, training and validation datasets were pulled together to obtain the input patches.
Ninety percent of image patches were used for training and ten percent of image patches were used for validation. U-Net. As the U-Net approach estimates a class at each pixel, three classes were defined to allow separating nuclei as proposed in  [19] : inner nuclei, nuclei contours and background. To facilitate nuclei separation, the nuclei contours in the training dataset were dilated  [19]. To reduce memory usage and limit over-fitting, the imaging field for images in the training dataset was set to 256 x 256 by randomly cropping the 512 x 512 input images. These cropped images were then normalized with a 1-99 quantile. A root mean square prop was used to estimate the parameters of the deep convolutional neural network by minimizing a weighted cross entropy loss to handle class imbalance for 100 epochs.
The weights associated with each class were defined from the training dataset as their inverse proportion. A data augmentation to increase the training dataset by a factor of 100 was processed before normalization with the imgaug python library  [48]  and included flipping, rotation, shearing, blurring, sharpness and brightness modifications, noise addition, intensity inversion and contrast modifications. Inception-V3. The same three classes were defined and the nuclei contours were dilated for the Inception-V3 approach as for the U-Net approach. The imaging field was set to 65 x 65 pixel image patches, large enough to include at least one nucleus as suggested in  [19]. For each image patch, only the class of the central pixel was trained or processed. The training images were normalized by dividing the intensity at each pixel by the median intensity over the image and by subtracting at each pixel the average intensity of a 65 x 65 neighborhood around it.
To obtain a balanced training dataset, the same number of input image patches were defined for each class. A stochastic gradient descent was used to estimate the parameters of the deep convolutional neural network by minimizing a categorical cross entropy for 15 epochs. A modification of the input image patches was processed after normalization with the imgaug python library  [48]  and included the same image transformations that were used for the U-Net approach. U-Net and Inception-V3 post-processing. For both U-Net and Inception-V3 approaches, nuclei were obtained by subtracting the nuclei contours from the inner nuclei. Let us define S c = {S c (x)} x2O where S c (x) denotes the nuclei contour score obtained with the U-Net or the Inception-V3 approach at pixel x, and O is the regular grid of pixels in the image.
5 for U-Net and 3. 5 for Inception-V3. From W, a set of K new nuclei separations fSðkÞg k¼1;. ;K is obtained. A score S w = {S w (k)} k=1,. ,K is defined for each new separation SðkÞ as follows: S w ðkÞ ¼ X x2SðkÞ S c ðxÞ À S n ðxÞ: ð3Þ All nuclei separations SðkÞ such that S w (k) > 0 are used to separate the nuclei obtained as the connected components of the binary image N b. Mask R-CNN. Version 2. 1 of Mask R-CNN  [31]  was used in this study. The backbone network was defined as the Resnet-101 deep convolutional neural network  [49]. We used the code in  [7]  to define the only class in this study, i. e. the nuclei. A data augmentation to increase the training dataset by a factor of 100 was processed before normalization with the imgaug python library  [48]  and included resizing, cropping, flipping, rotation, shearing, blurring, sharpness and brightness modifications, noise addition, intensity inversion and contrast modifications.
Transfer learning with fine-tuning from a network trained on the coco dataset  [33]  was also applied. In the first epoch, only the region proposal network, the classifier and mask heads were trained. The whole network was then trained for the next three epochs. Stardist. Version 0. 7. 1 of Stardist  [8]  was used in this study. The backbone network was defined as a U-Net  [27]. 512 x 512 input images were normalized with a 1-99 quantile. The data augmentation proposed by the authors was used to increase the training dataset. The whole network was trained for 400 epochs. Cellpose. Version 0. 6. 5 of Cellpose  [9]  was used in this study. The backbone network was defined as a U-Net  [27]. 512 x 512 input images were normalized with a 1-99 quantile. A data augmentation to increase the training dataset was processed before normalization with the imgaug python library  [48]  and included flipping, rotation, shearing, blurring, sharpness and brightness modifications, noise addition, intensity inversion and contrast modifications.
Training dataset As Mask R-CNN and U-Net are not suited for the task, the Inception-V3 method was used for marker identification. The training dataset consisted of eight confocal images for E2F3A, five confocal images for E2F8, four confocal images for EdU, three confocal images for pH3, twenty-one widefield images for E2F3A and fifteen widefield images for E2F8, E2F4, EdU and pH3, all annotated by a moderately-skilled researcher. Ninety percent of image patches were used for training and ten percent for validation. Two training strategies were designed to define the input image patches: The first only considered image patches centered at each nucleus center, while the second strategy included image patches centered at each pixel in the segmented nuclei, drastically increasing the amount of training data.
Inception-V3 Two classes (positive and negative) were defined for the E2Fs, while three classes (diffuse, punctate and negative) were defined for EdU and pH3. As for nuclei segmentation, the imaging field was set to 65 x 65 pixel image patches to ensure inclusion of at least one entire nucleus. The training image patches were normalized by subtracting at each pixel the average intensity in a 65 x 65 neighborhood. Intensities were not divided by the median intensity over the image as they were for nuclei segmentation because the median can be equal to zero for sparse markers such as pH3. To obtain a balanced training dataset, the same number of input image patches was defined for each class. A stochastic gradient descent was used to estimate the parameters of the deep convolutional neural network by minimizing a categorical cross entropy for 10 epochs.
To increase the size of the nuclei-based training dataset 100-fold, a data augmentation that included flipping, rotation, shearing, blurring, sharpness and brightness modifications, noise addition, inversion and contrast modifications was performed after normalization with the imgaug python library  [48]. A transfer learning with fine-tuning from the Inception-V3 network trained for nuclei segmentation on confocal and widefield images was performed on the pixel-based training dataset. In the first epoch, only the last layer of the network was trained. The whole network was then trained for the next five epochs.
ImageJ plugin Annotater We developed the plugin entitled Annotater to draw nuclear contours manually to define training datasets and to interactively correct deep learning-based nuclei segmentations. This plugin also provides for manual characterization of markers associated with nuclei and a thresholding tool for specifying an intensity threshold and the proportion of pixels in a nucleus above that threshold that defines a positive nucleus. Finally, Annotater can generate images in which segmented nuclei and marker characterization are overlaid on the input image and can extract nuclear features as defined in the Set Measurements window in ImageJ. The java code of the plugin is available at https://github. com/tpecot/Annotater/tree/master/src/main/java/ edu/musc/tsl and the plugin is available at https://github.
E2Fs accumulation over the cell cycle from 2D still images Data. As fluorescence intensity is proportional to protein concentration  [36, 37] , E2Fs nuclear accumulation over the cell cycle can be estimated from the intensity levels observed in the nuclei. To minimize the effects of noise while still being able to observe fluorescence fluctuations, we decided to define four levels of intensity, which is widely used in both diagnostic pathology and biomedical research  [38, 39]. In the previous section about marker identification, positive and negative cells were manually selected to define the training and evaluation datasets. By definition, negative cells do not express proteins and were consequently assigned an intensity equal to 0. Then, for each individual image, the average fluorescence intensity for each positive cell was processed and the range of average intensities from the lowest to the highest was binned into three to define levels 1 to 3.
Eq  (21)  takes into account the differences between E2F3A accumulation over the cell cycle when combined with any marker or any other E2F, with the same constraint as in Eq  (20). Finally, a spline fitting is applied to each of the E2Fs across all experiments to get a final accumulation over the cell cycle, which is shown in Fig 3. The code with the parameters used to estimate E2Fs accumulation over the cell cycle after initialization is available at https://github. com/tpecot/ EstimationOfProteinConcentrationOverTime. Simulated data. We generated simulations to evaluate the accuracy of the temporal estimation of protein concentration. For each simulation, the following concentrations are generated: y E2F3A j i 2 f0;. ; nb I g; i 2 f0;. ; nb T g; j 2 f0;. ; nb S g; y E2F8 j i 2 f0;. ; nb I g; i 2 f0;.
Combining confocal and widefield images in the training dataset improves the results, especially for confocal images (S4c-S4e Fig). This demonstrates how Mask R-CNN can benefit from an increase in training dataset size (from 4847/1619 nuclei in the training/validation dataset for confocal images alone to 15541/4051 nuclei in the training/validation dataset when confocal and widefield images are pooled together) even if the data are not uniform, coming from different sources. Of note, combined with massive data augmentation, transfer learning from the coco dataset really improves accuracy when considering confocal images only (S4e Fig compared to S4c Fig) or widefield images only (S4f Fig compared to S4d Fig), while it does not drastically improve the accuracy when confocal and widefield images are pooled together (S4c-S4f Fig).
Data availability All the images used in this study are available at https://data. mendeley. com/datasetsdatasets for nuclei segmentation are available at https://github. com/tpecot/ NucleiSegmentationAndMarkerIDentification/tree/ master/UNet/datasets/nucleiSegmentation_E2Fs for the U-Net architecture and at https://github. com/tpecot/NucleiSegmentationAndMarkerI Dentification/tree/master/MaskRCNN/datasets/ nucleiSegmentation_E2Fs for the Mask R-CNN architecture. The training datasets for nuclei segmentation and marker identification are available at https://github. com/tpecot/ NucleiSegmentationAndMarkerIDentification/tree/ master/InceptionV3/trainingData for the Inceptionare available at https://github. com/tpecot/ NucleiSegmentationAndMarkerIDentification. Archived code at time org/10. 5281/zenodo.
4619243 [50]  License: GPL3 The Octave code used to estimate the E2Fs accumulation over the cell cycle is available at https://github. com/tpecot/ EstimationOfProteinConcentrationOverTime. Archived code at time of publication: https://doi. org/10. 5281/zenodo. 4639800 [51]  License: GPL3 The Java code of the Annotater plugin and the plugin are available at https://github. com/tpecot/ Annotater. Video tutorials to show how to use the Annotater are available at the same location. Archived code at time of publication: https://doi. org/10. 5281/zenodo. 4639802 [52]  License: GPL3.
However, the utilization of this method of analysis quickly generates large quantities of data. With each tetraspanin detection antibody being used separately, 111 protein combinations are generated with six bead controls per sample. Analysis of this scale of data is currently limited by a lack of software analysis tools. Here, the use of stitched multiplex analysis, which provides a method to survey the expression of hundreds of protein combinations on EVs, is evaluated. Considerations for undertaking multiplex analysis for surveying EV repertoires are also outlined. The ability to perform stitched multiplex analysis to analyze data on this scale is demonstrated using a lab-built, open-source software package (multiplex analysis post-acquisition analysis software [MPA PASS ]) that allowed for high-throughput, multiparametric quality control and data analysis of EV multiplex data.
positive signals can have a better result compared with Miltenyi Biotec buffer alone. It is notable that the etraspanin signals CD9, CD63, and CD81 are all lower when samples are incubated in EV-depleted FBS, and this indicates that small EVs remain after the depletion method and may be able to compete with cell-culture-derived EVs to varying degrees with cross-reactive antibodies for both capture and detection. It is notable that one antibody, CD47, bound to every capture bead regardless of buffer. These data demonstrate the need to screen detection antibodies with capture beads before use. This is also plotted by ordering the average intensity across groups with the top 75 summarized in Figure  S4. For this reason, while including bead + antibody controls is essential, subtracting the bead + antibody intensity for bead + EVs +antibody is not recommended, asit may result in negative intensity data scaling.
Use of clustering to identify difference between biofluid and purification The use of multiplex analysis in sample cohorts can result in large arrays of data. Our proposed method of stitched multiplex analysis further increases that size. Methods of data reduction to identify populations in large datasets are beginning to become commonplace (e. g. , t-distributed stochastic neighbor embedding  [tSNE] ). With the accumulation of data across samples, equipment, assays, laboratories, and data reduction methods will play a role in understanding differences and similarities. For this reason, principal-component analysis (PCA) and tSNE have been built into MPA PASS. To understand the utility data reduction methods bring to stitched multiplex analysis plasma, serum, and CSF samples were analyzed using tetraspanins CD9, CD63, and CD81 individually at different input volumes and with different sample purification methods (Figure  4 ).
MPA PASS facilitates data sharing and atlas curation MPA PASS was developed in response to the lack of commercially available or free software packages for EV multiplex analysis. As demonstrated, MPA PASS provides researchers with normalization methods, multiplex data stitching capabilities, data reduction methods, and data visualization methods in open-source software. Along with a lack of software interfaces for EV multiplex analysis, there is also a current a lack of reporting standards for bead-based assays within the EV field. MPA PASS has attempted to facilitate this by being built on a standard database framework in the form of a spreadsheet containing key metadata criteria related to the assay (Table  2 ). This spreadsheet database framework (see key resources table) provides set sample, capture, and labeling metadata criteria for users to complete and allows users to further define their own metadata fields for stratification of their data analyses.
Databases can also be merged to create aggregate datasets, making it possible to create personal atlases. While all metadata are found within a spreadsheet, once imported into the software, all data and metadata are then stored in a single file in order to simplify data sharing. Metadata related to cytometer acquisition are already developed in the form of MIFlowCyt. Software for the calibration of data and the automated reporting of acquisition metadata has already been established using previously developed FCM PASS software.
The MPA PASS software has been developed for the future curation of datasets into the existing ERCC exRNA Atlas in mind (Figure  5 )  (Murillo et al. , 2019). The software is part of a package of tools developed for standardized reporting and analysis. While MPA PASS provides these tools for bead-based EV analyses, its counterpart, FCM PASS , provides standardized reporting and analysis tools for single-EV flow cytometry experiments, utilizing the MIFlow-Cyt-EV reporting framework  (Welsh et al. , 2020a (Welsh et al. , , 2020d, 2020e;, 2020e; Welsh and Jones, 2020). FCM PASS may also be used for '. fcs' file calibration prior to MPA PASS data import for fluorescence calibration  (Welsh and Jones, 2020). The use of these standardized reporting and analysis methods will aid in the transparency, reproducibility, and integration of EV repositories that will allow new approaches to EV characterization and subset identification.
One key consideration for the future of EV repertoire analysis with multiplex assays is that reproducibility and standardization across studies with these assays is an area for further development. A number of steps can be taken to improve assay quality and standardization: (1) quantification of the binding capacity of each bead in order to help standardize lots and Outlined are reporting metadata database fields generated upon MPA PASS dataset creation. Each criteria field is created within the spreadsheet and is organized onto sheets within the spreadsheet relating to sample, beads, labeling, or controls. The processing of single-EV data (1) using FCM PASS and bead-based assays and (2) using MPA PASS software produces standard reporting methods and data. MPA PASS data can be calibrated with FCM PASS (3) software or other means.
The product of these software packages is the integration of single-EV data (4) and bead-based EV data (5) that will enable standard comparisons and reporting for integration into online repositories such as the exRNA atlas. of a minimum set of reporting criteria based on established single-EV flow cytometry reporting frameworks  (Welsh et al. , 2020f). One of the largest hurdles of multiplex analysis optimization is resolving the nonspecific binding of detection antibodies to beads. Ideally, a positive control method allowing identification of EVs, rather than just a nonspecifically bound antibody, to the beads could be identified, such as using a membrane dye. Such positive control (marker-positive) EVs are not generally available, and, to date, the authors have been unable to identify a suitable membrane dye that is compatible with the multiplex beads used in this study.