dataesr/scientific-paragraphs-categorization huggingface dataset:
Single cell semantic segmentation Cell counting and segmentation is a common challenge in high-throughput analysis of optical microscopy images  [8, 9, 12, 33, 34]. Both fully-and weakly-supervised DL approaches were shown to be very powerful to assess these tasks on multiple cell lines  [7, 25]. We first highlight some prerequisite of the dataset to train MICRA-Net (and baselines) at solving an instance segmentation task using 6 selected cell lines from the Cell Tracking Challenge (CTC)  [8]. For weakly-supervised learning from image-level targets, a sufficient amount of negative samples (images not containing the object of interest) is required to extract informative context from an image, i. e. to distinguish the cells in the field of view. We trained MICRA-Net on 256 × 256 pixel crops from the resampled images of the CTC (with an effective pixel size of 0.
5 µm, Supplementary Tab. 6) and obtained a classification accuracy of (95. 8 ± 0. 4) % (calculated from 5 network instances). Despite having a high classification accuracy, MICRA-Net detection and segmentation performances were strongly reduced when no negative samples were provided (Supplementary Fig. As a proof of concept, using the CTC, for which precise annotations are available, we compared the semantic instance segmentation of MICRA-Net with fully-and weakly-supervised baselines: U-Net  [9] , Mask R-CNN  [10] , and Ilastik  [29]  (see Supplementary Note 3 for specific implementation details). The weak supervision consisted in dilating/eroding each object of the fully-supervised dataset by a value sampled from a normal distribution with 0 mean and standard deviation in {2, 5, 10} (Altered-X or ALT-X ), or by taking the bounding boxes of each objects (see Methods and Figure  4a ).
Since no precisely annotated testing dataset was provided for the CTC, we precisely annotated 4 images for each cell line to evaluate the segmentation performance of both approaches (precisely annotated dataset). We compared the achievable annotation precision from participants to that of altered versions of the precisely annotated dataset (Figure  4a, b ). Figure  4b  shows the distribution of IOU between associated objects (Object-IOU) of the User-Study (8 participants) and the altered versions of the dataset (8 repetitions) when comparing to the original precisely annotated dataset testing set for each selected cell lines of the CTC. From Figure  4b  we can conclude that the distribution of the User-Study is similar to the distribution of ALT-5. Hence, training a DL architecture with a training set obtained from multiple participants (e.
16 17 18 19 20 21 22 23 24 ). As shown in Figure  4d  and Supplementary Fig  24,  the performance of baselines which were developed for fully-supervised datasets is affected when reducing the supervision level (Supplementary Fig. 16 17 18 ). This is also depicted by the low classification accuracy of the baselines compared to MICRA-Net (Supplementary Tab. 7). Strikingly, MICRA-Net achieves cumulative similar performance to fully-supervised Mask R-CNN and Ilastik for the semantic instance segmentation on the 5 cell lines. MICRA-Net is only significantly outperformed by U-Net when training is performed on the fully-supervised or a slightly altered (ALT-2) dataset. Therefore, when no precisely annotated and proofed dataset is available, or when the annotation error may be high, the performance of baseline architectures cannot be guaranteed to achieve superior semantic instance segmentation performance on all cell lines (see Supplementary Fig.
24 , and Supplementary Tab. 8  and 9 ). The performance of the conducted User-Study on the testing dataset were also compared to MICRA-Net (Figure  4d  (right), Supplementary Fig. 2, 3 ). A significant increase in performance is measured for MICRA-Net for bounding boxes and no significant change is observed when comparing to precise annotations. Given the previous results, an approach like MICRA-Net will perform similarly (or better) to the presented baselines for semantic instance segmentation when no precisely annotated dataset is available. More importantly, MICRA-Net reduced by a factor of 40 the number of Expert decisions required to annotate the training dataset and by more than 150h the necessary annotation time usually needed to complete this task while achieving precise human-level precision (Figure  1f  and Figure  4c ).
Multi-device analysis While DL approaches can be very powerful when tackling tasks on very similar images, challenges are often encountered when the imaging conditions change over time (e. g. due to a new device)  [37, 38]. To increase the applicability of the proposed method to various experimental conditions, we investigated how MICRA-Net could be fine-tuned on a new dataset that contains similar structures but acquired on a new device. To address this, a brightfield microscopy dataset of Giemsa-stained  [39]  P. Vivax (malaria) infected human blood smears was used (Figure  5a ), for which the training and testing datasets had very distinct intensity distributions (Figure  5a, b )  [33, 36]. The first attempt to solve the classification task consisted in predicting the presence of infected smears in a 256 × 256 pixel image.
A testing classification accuracy over 87% was obtained when updating Without fine-tuning, the performance on the testing dataset (Naive, grey ellipse) is characterized by a recall of 0. 79, and a poor precision of 0. 32. A variable number of images ({12, 24, 36}) from the testing dataset were used to adjust the detection threshold (Threshold, blue ellipse), which increased the precision but also reduced the recall by approximately 2 folds. Fine-tuning of the model on the sampled {12, 24, 36} images from the testing set with different settings: i) allowing the linear layer (orange), and ii) different depths (depth 4: green; depth 3, 4: red) to be updated (see Supplementary Fig. 25  & Supplementary Note 4) resulted in precision-recall above human level detection. c, right) Zoomed region of the precision-recall performance of MICRA-Net.
MICRA-Net, a CNNbased method, relies on the information embedded in the latent space of a main simple task, in our case classification, to learn multiple complementary tasks without the need to generate task-specific precisely annotated training sets. We designed multiple experiments to challenge MICRA-Net at solving common microscopy tasks (segmentation, enumeration, or localization) relevant to high-throughput microscopy image analysis  [3, 9]. Unlike multi-task learning  [43] , MICRA-Net does not combine auxiliary tasks to increase the learning performance of a main task, nor requires more annotations from the dataset for each task  [44, 45]. Hence, the use of MICRA-Net should significantly reduce the burden of task-specific annotation of bioimaging datasets thereby increasing the accessibility of such deep learning based microscopy image analysis.
Our results show that MICRA-Net can be applied to various microscopy modalities and biological contexts, while significantly reducing the number of required Expert decisions to generate the training dataset (Figure  1b ). While fully-supervised DL approaches (e. g. based on U-Net or Mask R-CNN architectures) have the drawback of being costly to train, they can benefit from pre-training  [9, 46, 47]  given the image space is similar  [48] , and have access to precise information about the structure boundaries. On the other hand, MICRA-Net leverages on the extraction of spatial features from the hidden layers of the network to generate detailed feature maps using solely, easy to retrieve, binary image-level annotations for training. Considering the observed reduction of the inter-expert variability when diminishing the complexity of the annotations, this will be an important aspect for future applications leveraging on crowd-sourced annotations for training.
MICRA-Net provides similar or even superior performance on multiple tasks to the state-of-the art weaklyand fully-supervised learning approaches, thus making it an unprecedented alternative to address bioimaging analysis challenges for which large and precisely annotated datasets are not available. Additionally we demonstrated that MICRA-Net could be fine-tuned when facing strong variations in the quality of the available datasets, for example when images were acquired on two different microscopes. Fine-tuning of the architecture on few images from another microscopy system was sufficient to achieve better detection efficiency than inter-expert agreement. This is of particular interest for large-scale studies, conducted on multiple sites, that require analysis framework to be easily adaptable to new experimental conditions  [33, 49, 50].
This allowed an increase in the total number of detected regions of interest (genetically-tagged axons) by 25% while reducing the required annotation time for newly acquired images by 30 folds. Precise annotations, even if obtained from trained Experts, are associated with inter-participant variability, especially when defining the boundaries (Figure  1 ). This variability needs to be assessed to characterize the annotated dataset and the precision of the neural network precision  [3, 51]. We observed that image-level binary annotations can help to increase the consistency among Experts by reducing the complexity of the annotation task. By alleviating the annotation burden, an approach such as MICRA-Net can help increasing the accessibility of deep learning assisted quantitative image analysis in microscopy.
As a whole, it can be used in multi-class detection, segmentation, counting, and classification tasks in bioimaging, for which a precisely annotated dataset is not available or tedious to obtain. Figure  2a  shows the schematic representation of the MICRA-Net architecture. MICRA-Net is based on the encoder part of a U-Net  [31]. The rationale is that U-Net is an established method to solve different analysis tasks (e. g. segmentation, localization, detection) on biomedical datasets. Each depth of the network contains two blocks of convolutions (kernel size of 3) followed by batch normalization, and ReLU activation. The number of filters in the convolutional layers is doubled after maxpooling (stride and kernel size of 2) to increase the richness of the representation. The number of filters for each layer is {32, 64, 128, 256}.
Training procedure The general training procedure of the MICRA-Net architecture are reported within this section. For specific training details for each dataset, see Supplementary Notes 1-5. MICRA-Net was trained using the Adam optimizer with a learning rate specific to each dataset and other default parameters  [54]. A learning rate scheduler was used to reduce the learning rate of the optimizer with a minimal possible learning rate of 1 × 10 -5. The number of training epochs was adapted to the specific dataset (Supplementary Tab. [22] [23] [24] [25] [26]. Early stopping was used to reduce overfitting. Unless otherwise specified, we used binary cross entropy with logits loss. We kept the model with the best generalization properties on the validation set (calculated from the objective loss function).
Auxiliary tasks This section presents how MICRA-Net can be used to solve the common auxiliary tasks in microscopy images. Classification. The classification task is used on all presented dataset in the paper. It serves as a guideline to validate the representation capability of MICRA-Net. The classification task is solved by design using MICRA-Net since it is trained using a classification task. The prediction from MICRA-Net are mapped in the [0, 1] range using a sigmoid function. Semantic segmentation. The semantic segmentation task is solved on all presented dataset in the paper. This task is solved by first extracting a detailed semantic feature map as described in Section 4. 1. 1. The semantic segmentation masks are obtained by thresholding the resultant semantic feature map using common thresholding algorithm (e.
g. Otsu or percentile thresholding). The dataset specific thresholding is detailed in Supplementary Notes 1-5. Detection. The detection task on the P. Vivax and EM microscopy dataset is solved by predicting the probability of presence of an object on all extracted crops. The overlap between the crops is of 75% in both directions. Overlapping crops are averaged and reassigned to an output feature map of the same shape as the image. The detection threshold is inferred from the validation set using a precision-recall curve. Semantic instance segmentation. The semantic instance segmentation task is required on the Cell Tracking Challenge dataset. MICRA-Net is required to predict i) the presence of an object and ii) the contact between objects. The grad-CAMs of the activated objects are extracted from the architecture and combined using a principal component analysis (PCA) as presented in Section 4.
1. 1. If a contact is predicted on an image, the grad-CAM from L 8 which contains the prominent information of the contact is extracted. The contact feature map is subtracted from the object feature map as in some fully-supervised techniques  [27]. An Otsu threshold is used to generate the semantic segmentation masks of the instances. The modified MNIST testing dataset consists of 1000 images of handwritten digits sampled from the original MNIST testing dataset. As for the training dataset, we also applied, in a random order, Gaussian blur and artificial normalized Poisson noise sampled as before.
Datasets by an Expert. The task was to differentiate infected from uninfected blood smears. The dataset is highly unbalanced towards red blood cells which composes over 95% of the annotated cells. For training and testing, we applied a whitening normalization (null mean and standard deviation of 1) to each image (and channel) to minimize the impact of a very different intensity distribution. The binary targets for training were generated using the provided bounding boxes. A crop was considered as positive if it contained at least 5% of overlap with an infected cell, otherwise as negative. The crops were 256 × 256 pixel. We manually extracted and precisely annotated all infected cells in the testing set resulting in 303 small crops of size 256 × 256 pixel centered on the cell of interest.
Scanning Electron Microscopy dataset The dataset contained 92 images of 10, 240 × 10, 240 pixel for training, 66 for validation, and 44 for testing. An Expert annotated the images using positional markers to locate the Axon DAB markers. On average the large fields of view contained 3 small detections (113 × 113 pixel, between 1 and 10 detections per image). All Axon DAB markers were extracted from the testing set (170 positive markers) and an Expert carefully identified their contours.
Detection The centroid of each detected object was obtained from MICRA-Net by using the dataset specific procedures detailed in Supplementary Notes 1-5. Each detected centroid was associated with the centroid of objects in the ground truth mask using the Hungarian algorithm  [57]  with a maximal distance of N pixels, where N is approximately the object radius. In this context, an associated detected object is considered as a true positive, a non-associated detected object is a false positive, and a missed ground truth object is a false negative. To evaluate the detection capability of MICRA-Net, we reported the F1-score. For a quantitative comparison, we repeated the evaluation for each trained model. We then bootstrapped the average of the trained models to show the bootstrapped mean and 95% confidence interval (10 000 repetitions).
Tissue preparation for scanning electron microscopy dataset After a period of 21 days following stereotaxic injection, mice were anesthetized with a mixture of ketamine (100 mg/kg) and xylazine (10 mg/kg) and transcardially perfused with 50 ml of phosphate-buffered-saline (PBS: 50 mM at pH 7. 4) followed by 150 ml of 4% paraformaldehyde (PFA) and 1% glutaraldehyde diluted in phosphate buffer (PB; 100 mM at pH 7. 4). Brains were dissected out, post-fixed for 24h in the same fixative solution and cut with a vibratome (model VT1200; Leica, Germany) into 50 µm-thick frontal sections, which were serially collected in sodium phosphate buffer saline (PBS, 100 mM, pH 7. 4). Frontal brain sections at the level of the subthalamic nucleus (STN) were processed to reveal the presence of APEX2 in axons arising from DRN-infected neurons using 3,3'diaminobenzidine (DAB; catalog no.
The F1-score for points annotation (green) was calculated with a maximal distance of association of 30 pixels. Plotted are the bootstrapped mean (line) and 95% confidence interval (shade, 10 000 repetitions). c-e) Shown are the distribution of median scores from the inter-participant comparison calculated in a all versus one manner. c) Distance between associated point markers. d) Average distance between the precise contours of participants annotations was calculated for precise annotations. e) Average required time per objects on different cell lines for each supervision level. f) Evaluation of the annotation task required to generate the training set for all microscopy datasets used throughout the paper for fully-supervised (FS) and MICRA-Net approaches. Reported above is the effective number of decisions (number of extracted crops for MICRA-Net and number of edge pixels for fully-supervised learning) and the required time in hours.
Each activated class is backpropagated through the network and a local map for each layer of the network (L 1-8 ) is computed. See Supplementary Fig. 5 for layer specific grad-CAMs. d) Detailed segmentation maps of the digits of a representative image (256 × 256 pixel) and insets (right, dashed white box) from the modified MNIST dataset using MICRA-Net. The color code corresponds to the digit class and the red arrow indicates a missed digit in the field of view. e) Mean performance over the 10 classes obtained with the U-Net trained with and without dilation of the ground truth contours. The segmentation maps are presented in Supplementary Fig. 7a. MICRA-Net segmentation performance (color-coded dashed lines, see Supplementary Fig. 5 for distributions) surpasses the U-Net trained with 10 pixels dilation and is not statistically different from the U-Net trained with 5 pixels dilation on all measured metrics.
13 , DIC-C2DH-HeLa and Fluo-N2DH-GOWT1). It is therefore necessary to adapt the size of the training images that are provided to the network to the size of the structures of interest, ensuring that enough images contain only background (Supplementary Tab. 6 for selected factors). Another requirement when training a deep learning architecture is that the object of interest can fit entirely within the field of view. Otherwise the model has no information on how different parts of an object should be tied together. To reflect this statement, we trained both U-Net and Mask R-CNN on a resized version of the CTC dataset containing positive and negative samples on all cell lines (Supplementary Fig. 14and Supplementary Tab. 6 for scale factors). We observe that the performance of all models is significantly lower on the DIC-C2DH-HeLa cell line at this scale.
16-18 for baseline examples. d, left) We compared the difference of the pooled area under the curve (AUC, F1-score vs. IOU) of all cell lines for MICRA-Net over the baselines on the precisely annotated dataset. The raw curves are available in Supplementary Fig. 19-23, and the non-pooled data in Supplementary Fig. 24. Higher and lower performance of MICRA-Net are reported in blue and red respectively. MICRA-Net is only outperformed by U-Net trained using ALT-2 or fully-supervised training. d, right) We compared the pooled AUC for all cell lines for the conducted User-Study using precise annotations and bounding boxes. The precision of the segmentation masks generated with MICRA-Net is similar to the precise annotations and better than the bounding boxes obtained in the User-Study. Stars are used to highlight a significant change (Supplementary Tab.
79, and a poor precision of 0. 32. A variable number of images ({12, 24, 36}) from the testing dataset were used to adjust the detection threshold (Threshold, blue ellipse), which increased the precision but also reduced the recall by approximately 2 folds. Fine-tuning of the model on the sampled {12, 24, 36} images from the testing set with different settings: i) allowing the linear layer (orange), and ii) different depths (depth 4: green; depth 3, 4: red) to be updated (see Supplementary Fig. 25& Supplementary Note 4) resulted in precision-recall above human level detection. c, right) Zoomed region of the precision-recall performance of MICRA-Net. When the number of trainable parameters increases, the number of images required for a model with good generalization properties also increases.
9. The annotated area as a function of the recall for each PU ratio is shown in Supplementary Fig. 27. c) Total number of detections from the testing dataset with and without assistance from MICRA-Net. Using MICRA-Net the Expert could identify 57 new Axon DAB positive regions which correspond to an increase of 25% in the total number of detections. The scale bar is 5 µm for the full field of view and and is 1 µm for extracted crops.
Modified MNIST datasetWe generated the modified MNIST training dataset by randomly sampling N digits from the original MNIST training dataset and randomly distributed them on a 256 × 256 pixel field of view. To avoid overlap between digits we used a random Poisson disc sampling algorithm with a radius size of 25 pixels [55]. The number of digits N was uniformly sampled from {1, 2, 3, 4, 5, 10, 15, 20, Max}, where Max corresponds to the maximum number of digits that can be placed without overlap. A rotation of ±30 • uniformly sampled was applied to the digits before placement on the image. We applied, in a random order, a Gaussian blur with sigma uniformly sampled in [0, 2[ and artificial normalized Poisson noise with λ = √ 255 The resulting image intensities were clipped to lie in [0, 1]. Using this technique, we generated 2000 and 1000 images for training and validation respectively.
Pixel size was set to 20 nm. 4. 5. 2 Animals and stereotaxic injections for scanning electron microscopy datasetThis study was carried out on 3-month-old mice, weighing 25-35g. Animals were housed under a 12h lightdark cycle with water and food ad libitum. All procedures were approved by the Comité de Protection des Animaux de l'Université Laval, in accordance with the Canadian Council on Animal Care's Guide to the Care and Use of Experimental Animals (Ed2), and with the ARRIVE guidelines. Maximum efforts were made to minimize the number of animals used. Transgenic e-Pet Cre mice expressing Cre recombinase under the control of Fev promoter, known to be specific for serotonin (5-HT) neurons [59] , were injected in the dorsal raphe nucleus (DRN) with 1 µl of AAV9-CAG-DIO-APEX2NES-WPRE. Stereotaxic injections were done using a 30 • angle along the frontal plane at AP: -4.
This resulted in an annotation time of approximately 30 minutes per field of view. Training and inference was performed on 512 × 512 pixel size crops. The dataset contained all positive crops (1024 × 1024 pixel, centered on the Axon DAB markers), and all negative crops (without overlap). To manually annotate the images the Expert inverted the acquired images. Hence, we provided MICRA-Net with the inverted image to mimic the Expert task. We rescaled the provided 8-bit depth images in the [0, 1] range by dividing by a scalar value of 255.