dataesr/scientific-paragraphs-categorization huggingface dataset:
We iterated ground truth annotation and computation of intermediate results until the prediction quality started to converge. We observed that this targeted selection of ground truth cubes can improve both the speed and quality of the entire segmentation process. As a rule of thumb, a total of 3 annotated cubes was a good starting point with an increase of 2 new cubes in each iteration. Note that this strategy should be adapted to data size, diversity within the segmentation class, and quality of the intermediate segmentation results. CebraEM capitalizes on a readily trained boundary prediction CNN (CebraNET). We observed that, when trained for boundaries, a model generalizes well and reliably picks cellular membranes. Membranes appear very similar in most volume SEM datasets, regardless of biological or experimental variability or sample preparation variations, and thus enabled the prediction of membranes without the need for retraining on new target data (see section "Transferable membrane prediction").
By doing this, the model not only learns to connect the target organelle supervoxels but also to differentiate it from the background and adjacent objects, even if they are of the same class. Consequently, even if the segmented objects are in contact, this training step can result in a semantic instance model. After the model is trained and validated on the provided ground truth cubes, the full dataset can be segmented by inference of the model to the entire volume (Fig. 1c ). At any time, from initialization to the final segmentations, the CebraEM project with its associated raw data, membrane prediction map (CebraNET), supervoxel map, and segmentation maps can be viewed using the MoBIE viewer  (39). Transferable Membrane Prediction. As described in the introduction, one of the key aspects of segmentation is to split the dataset into separated regions that accurately delimit the organelles' boundaries.
Since the sample preparation for the volume EM dataset is generally tuned to make cell membranes visible, we trained a CNN to generate membrane predictions (CebraNET) as a basis for this initial partition. First, a section of 512 x 512 x 64 voxels was selected from a publicly available FIB-SEM dataset (EMPIAR-10311) which contained a high number of cellular organelles. This section was densely annotated for organelles using Microscopy Image Browser  (40). From this organelle segmentation, we derived object boundaries, which we defined as pixels located at a change of object label in the segmentation. Note that we masked out the inner part of organelles in the model training process. The generated boundary ground truth, in conjunction with masked organelle interior, was used to train a concatenation of three 3D U-Nets  (23)  in order to predict organelle boundaries in volume SEM data (Fig.
2a ). For training, we used a binary cross entropy loss applied to the output of each successive U-Net module thus comparing the output to the organelle boundary map. Additionally, we combined different types of data augmentation (including brightness and contrast variations) to achieve a better generalization of the model and successfully predict dim boundaries in the test datasets. The final model, which we called CebraNET, can be seen as a dedicated boundary predictor for volume SEM datasets, predicting plane-like structures like cellular membranes. The predicted membrane edges delimit separated regions, which enabled a good initial partitioning into supervoxels. This is a crucial step since the subsequent steps require a decisionmaking process to join those supervoxels into full objects.
In practice, when applied to a different dataset, CebraNET was able to infer membrane-like structures, even if they were dissimilar to the original dataset (Fig. 2b  and c ). In the HeLa-1 and HeLa-2 datasets, membranes of a broad variety of organelles were successfully predicted including innerorganelle membranes, with the input data scaled to a resolution of 5 nm isotropic for the Hela-1 dataset (Fig. 2b ) and the native 4 nm resolution for the Hela-2 dataset (Fig. 2c-i ). For the Calu-3 dataset, organelle boundaries of mitochondria and DMVs were well predicted at the native 8 nm isotropic resolution (Fig. 2c  ii). However, compared to the HeLa datasets, the ER was not well reconstructed. An artificial increase of the resolution to 5 nm isotropic yielded a good boundary reconstruction such that the segmentation of ER in this dataset became feasible (Supplementary Figure  S1 ).
In contrast, the Macrophage-4 dataset, in its native 5 nm resolution, yielded a boundary prediction with a high level of detail, such that the segmentation of the ER was feasible in its native resolution (Fig. 2c  iii). The Platynereis-5 datasets showed the strongest anisotropy of the tested datasets as well as the lowest xy-resolution. For the segmentation of mitochondria, we mapped the voxels of 10 x 10 x 25 nm 3 anisotropic input data to an isotropic resolution of 10 x 10 x 10 nm 3. In this way, we improved the ac-curacy of the prediction for mitochondrial outer membranes and enabled the downstream steps for this dataset (Fig. 2c iv ). Overall, using different input resolutions to the network appeared to be an efficient method to target boundaries of a certain thickness and distance. Close membranes, such as the outer double membranes of mitochondria and DMVs could be well predicted as a single boundary when using a low resolution (supplementary Figure  S1 c-ii  and d-ii ).
The resulting segmentation may contain over or under-segmentation and is not specific to any targeted object (Fig. 1b-i ). The annotator can correct the segmentation by merging or splitting off supervoxels (Fig. 1b-ii ) and review the result until all objects are considered correctly segmented. Alongside, the annotator can tag each segmented object to its respective class, hence, achieving semantic instance segmentation (Fig. 1b-iii ). We performed annotations on cubes of approximately 256 x 256 x 256 voxels. This size enabled a good trade-off between an overview of the data at hand, where the volume section is still large enough to include substantial amounts of the target structures and resulting annotation times. Annotation times lay mostly within the range of approximately 15 to 30 minutes for each dataset and varied strongly depending on the complexity of the target organelle morphology and the number of instances present in the annotated cube.
Therefore, the required amount of ground truth for training a data-specific The input of the model was the FIB-SEM data, and the output was the annotated outer membranes of organelles in form of a white mask, with the areas inside organelles masked (gray), and the background in black, ignored. The loss function was computed as the difference between masked annotations and the output of the network as a probability map of being a membrane. b) After training, the same model is applied to new raw FIB-SEM data, resulting in a prediction of almost all cellular membranes including those which have not been used during the training (e. g. within organelles such as the mitochondrial cristae). c) The inference model was successfully applied to other datasets, displayed here with an increasing difference to the original training dataset: (i) HeLa-2, another Hela cell, also processed using high-pressure freezing and imaged with the same microscope type, a FIB-SEM, (ii) Calu3-3, which is a human lung cancer cell, with different treatment, chemical fixation, and same microscope type (FIB-SEM), (iii) the Macrophage-4 dataset, from a chemically fixed cell imaged with a FIB-SEM and, finally (iv) Platynereis-5, from a chemically fixed multicellular annelid, imaged with an SBF-SEM.
Scale bars: 0. 5 µm. model could be generated in a few working days (Fig. 3  and supplementary tables S1. 1, S2. 1, S3. 1, S4. 1, and S5. 1). In order to maximize the efficiency of annotation, we used the most suitable resolution for each organelle. Organelles such as ER, NE, and the Golgi apparatus were annotated at a resolution of 4 to 5 nm isotropic to be able to include all their fine structural details (see supplementary figures S1a and b for an illustration using the Calu-3 dataset). For larger organelles, such as mitochondria or double-membrane vesicles (DMVs), we used 8 or 10 nm isotropic thus increasing cellular context in computations and increasing computation speed while the supervoxels boundaries still defined the organelle instances well (supplementary figure  S1c  and d ). In practice, within one dataset, we grouped organelles of similar size scale (e.
g. ER, NE, and Golgi apparatus or mitochondria and DMVs) to be able to jointly annotate them and thus additionally increase the efficiency of the annotation process. More specifically, the Hela-1 dataset (native voxel size at 5 x 5 x 8 nm 3 ) was annotated for ER (11 cubes), NE (6 cubes), and Golgi (6 cubes) using an up-sampled 5 nm isotropic resolution as well as mitochondria (4 cubes) using 10 nm isotropic resolution (Fig. 3a ). The annotation time for the ER, NE, and Golgi in the Hela-1 dataset was 12. 8 ±6. 2 (SD) minutes per cube covering a total cellular volume of 31 µm3. For mitochondria, the annotation time was 11. 5 +-1. 9 (SD) minutes per cube covering a total volume of 67 µm3 (Fig. 3a  and Supplementary Table  S1. 1). For the other datasets, i. e. , the Hela-2, Calu3-3, Macrophage-4, and Platynereis-5 datasets, we proceeded accordingly (Fig.
3b  to d and Supplementary Tables S2. 1 -S5. 1). Organelles of similar size scale were grouped to an equal annotation resolution (e. g. , DMVs and mitochondria in the Calu-3 dataset were annotated using 8 nm isotropic resolution). For finer structures such as the ER, we annotated at higher resolution (4 to 6 nm), thus enabling a more detailed membrane probability map that resulted in finer supervoxels (e. g. , the T. gondii and macrophage ER were annotated using 4 and 5 nm isotropic resolutions, respectively). To guide a user most efficiently through the process of data annotation, we implemented a plugin for the Napari viewer (supplementary Figure  S7 ). An extracted ground truth cube containing the raw data, the pre-computed membrane prediction, and the supervoxel maps can be loaded and visualized in Napari (Fig.
Both random forest models, the edge and the semantic model, are trained using features derived from the annotated subsets. The feature space is set up from convolutional and statistical features extracted from the raw volume, the membrane prediction map, and the supervoxels. They comprise edge features (intensity statistics along supervoxel edges on the raw data and membrane map), region features (intensity statistics within each supervoxel from the raw data), and shape features (supervoxel shape measures). The edge model is trained using all computed features while the semantic model is trained exclusively on region and shape features (Fig. 4a-i ). For inference, the full target dataset is processed as evenly spaced blocks, processed individually with a certain overlap (32 pixels in each dimension) to avoid boundary issues.
First, for each block, features are extracted from raw images, membrane, and supervoxel maps as described for the model training. Second, the inference results of the edge and semantic model are obtained. The edge probabilities are converted to edge weights and passed to the multicut algorithm, which solves the multicut objective for each of the blocks, generating an instance segmentation. Finally, classes are assigned to each instance using the semantic model output. The class of each individual object is defined by a majority vote of its supervoxels, where each supervoxels votes for its most likely associated class. The results for all blocks are stitched back to the full volume by greedily merging organelle instances at block boundaries, if they intersect by more than 50 percent, thus achieving a consistent segmentation for the entire dataset (Fig.
4a-ii ). To examine the versatility of our approach we performed a segmentation of organelles for multiple datasets. In the Hela-1 dataset, we segmented ER, Golgi apparatus, mitochondria, and nuclear envelope (Fig. 5a  and b ), and in the Hela-2 dataset we segmented ER (including the nuclear envelope) and mitochondria (Fig. 4b-i ). The Calu3-3 dataset was segmented for ER, mitochondria, and SARS-Cov2-induced double-membrane vesicles (DMVs) (Fig. 4b-ii ). For the Macrophage-4 dataset, we segmented the ER from both the macrophage host and the internal parasite T. gondii (Fig. 4b iii). Finally, we challenged the workflow with a very large SBF-SEM dataset, by segmenting the Platynereis-5 dataset for densely packed mitochondria in ciliated cells (Fig. 4b-iv ). This automated pipeline yielded proper semantic instance segmentations that did not need further post-processing.
Furthermore, for average-sized datasets typically obtained from a FIB-SEM session (e. g. 16,000 µm 3 at 8 x 8 x 8 nm 3 voxel size, 3 days acquisition on an off-the-shelf FIB-SEM), results were obtained within two weeks, from the generation of the Most annotations, even when multiple organelles were jointly annotated, could be finished in less than 30 minutes for a non-expert user of the CebraANN workflow. In every image, the time needed to complete the annotation is displayed at the bottom left of the 3D rendering. The input data volumes were scaled to isotropic voxel sizes in different resolutions to optimize annotation accuracy and annotation speed. Cubes of lower resolution (e. g. , 10 nm) can be annotated much faster with respect to the sample volume they cover, however, do not yield enough information to successfully annotate more detailed organelles such as the ER or Golgi apparatus.
Evaluation of accuracy. To evaluate the quality of the automated segmentations, we annotated additional validation cubes for each dataset and organelle using the CebraANN workflow. For each dataset, we selected extra cubes at random locations. We ensured that each cube that was used in the validation contained the desired target organelle and did not overlap with any of the training cubes. Then, we compared the automated segmentation result with the validation data using the ratio of correctly segmented areas against the union of true positive and false positive segmented areas (also known as IoU, Intersection over Union) (Table  1  For example, for the Hela-1 dataset, our segmentations were validated with 8 cubes containing ER, 3 cubes containing NE, 3 cubes containing Golgi apparatus, and 3 cubes that contained mitochondria.
Since ER, NE, as well as Golgi, were annotated at the same resolution level (5 nm isotropic), we used the same cube for multiple organelles when applicable. The segmentations for mitochondria provided an IoU of 0. 96 ±0. 04 (SD) (Fig. 5f-iii ) and an ER of 0. 93 ±0. 08 (SD) (Fig. 5c  iii). The other organelles had values of 0. 73 ±0. 17 (SD) (Fig. 5d -iii) for NE and 0. 65 ±0. 04 (SD) (Figure  5e iii) for the Golgi apparatus. Note that in the Golgi apparatus, most IoU errors originated from false negatives around a well-detected inner core of the respective Golgi stack, indicating ambiguities in the assignment of objects as either   vesicles or part of the Golgi stack. However, all Golgi stacks present in the full volume were found by the prediction (Table  1 , row Hela-1). The segmentations of the other four datasets (Hela-2, Calu3-3, Macrophage-4, and Platinereys-5) were evaluated in the same way.
We used a minimum of three validation cubes per organelle to obtain reliable evaluation metrics. Our approach achieved IoU scores of 0. 7 or higher, mostly depending on the difficulty of the respective organelle. For example, ER in the Calu3-3 dataset yielded a final IoU of 0. 78 ±0. 2 (SD) while DMVs in the same dataset achieved an IoU of 0. 99 ±0. 01. In many cases, we were able to achieve segmentation of IoU > 0. 8 after two iterations of annotations (supplementary tables S6 to S10). The first iteration establishes an initial set of ground truth cubes and the second iteration adds more information at locations where the initial model underperformed. This was the case for the segmentations of both ER types (macrophage and T. gondii) in the macrophage dataset with an IoU of 0. 84 ±0. 04 (SD) and 0.
93 ±0. 06 (SD), respectively, as well as for the segmentation of mitochondria in the Platynereis-5 dataset with an IoU of 0. 82 +-0. 14 (Table  1 ). Since the data partitioning was based on a 3D boundary prediction, the object boundaries of the generated organelle instances showed smooth and plausible appearances in all three spatial dimensions. However, the segmentation quality still depended on image quality as well as some morphological aspects of the target organelles. Small objects (only a few pixels in width) were often not resolvable, as there was no significant space between the predicted membranes to form suitable supervoxels in the downstream processing. This situation arose, for example, for narrow stretches in the ER as found in the Hela-2 dataset (Supplementary Fig. S8 ). In other cases, two significant membranes could be recovered by increasing the resolution artificially (e.
g. from 8 nm to 5 nm as used in the Calu-3 dataset, supplementary Figures  S1b  and c ). The same effect, i. e. a missing space in-between two membranes, could be observed at inter-organelle cytoplasmic space, observed for example, at some mitochondria-ER contact sites (Fig. 5b , Hela-1) or adjacent Golgi cisternae (Fig. 4b-i ). Consequently, thin objects or thin areas of larger structures were merged into the surrounding region, usually the cytoplasm, reducing the overall accuracy slightly.
Discussion Segmentation of volume SEM datasets is a challenging task with data-dependent difficulties originating from sam-ple type, sample preparation, biological perturbations (e. g. , knock-downs, drug effects, or virus infections), microscope hardware, image acquisition parameters, and system stability. With such a diversity of variables affecting the final data appearance, training a generic organelle segmentation model or using unsupervised approaches still remains a difficult task. With CebraEM, we followed a strategy that prioritizes the supervised training of shallow learning models on the target data, whilst paying specific attention to the time budget necessary to achieve a satisfying semantic instance segmentation. As a result, complex segmentation tasks are achieved single-handedly within a reasonable amount of time (a few working days), even on dense volumes.
The basis for CebraEM's efficiency and generalization to the broad variability of volume SEM datasets is our readily trained membrane model, CebraNET. The CebraNET membrane model can be adapted to the detail level of the respective target organelle by variation of the pixel size of its input, which thus forms the input pixel size to the CebraEM workflow. Within this work, we are showing CebraEMs segmentation capabilities within a resolution range from 4 to 10 nm, yielding a high level of complexity in organelles such as the ER, NE, or the Golgi apparatus, while increasing efficiency for organelles where a high-resolution level is not required (e. g. mitochondria or DMVs). We additionally demonstrate the successful application of CebraEM to multiple datasets originating from different imaging modalities (FIB-SEM and SBF-SEM), sample processing techniques (high-pressure freezing in conjunction with freeze substitution or chemical fixation), and sample types.
In some cases, thin objects, such as very narrow ER stretches or small cytoplasmic spaces between adjacent organelles, collapse the membrane model and thus inhibit supervoxel generation. These areas then are not detectable even in the manually guided CebraANN workflow. However, we believe, for a general-purpose tool and with the goal of maximum efficiency, this is an acceptable cost. Also, a subsequent deep learning-based approach to detect and fix such instances could be implemented. The entire CebraEM workflow is available as an open-source package on GitHub (github. com/jhennies/CebraEM), implemented in a script-based fashion for usage on a workstation or cluster setup. The embedded annotation tool CebraANN is a graphical user interface implemented as a plugin for the image analysis software Napari.
Overall, the proposed Ce-braEM workflow is designed to enable segmentation in large datasets without the requirement for extensive expertise in machine learning while reducing the amount of manual input. We have the conviction that CebraEM will contribute to generating faster and better 3D models for the volume SEM community. Additionally, the annotation tool CebraANN in conjunction with CebraNET could be used for other applications including the efficient generation of ground truth for deep learning models or the segmentation of entire small datasets.
Data visualization and data storage. The big-dataviewer-based MoBIE viewer  (39)  was used to inspect large datasets. MoBIE enables image pyramid-based lazy loading of the data and scales to arbitrary data size. Additionally, it can display multiple datasets such as raw data, processing steps, and final segmentations in an overlay representation. We implemented CebraEM projects in a MoBIEreadable format such that they can be directly viewed without any conversion requirements. Before initialization of the CebraEM project, the raw data has to be converted to the n5based BigDataViewer  (43)  format for which CebraEM ships a conversion script. All computational results (membrane prediction, supervoxels, and segmentation maps) are, on the fly, saved into the project structure in the required n5-based BigDataViewer format which enables parallel writing for increased efficiency of data I/O.
Already computed subsets can then be visualized in the MoBIE viewer already before the completion of the computation. Preprocessing. We use AMST  (41)  to align each of the volume image stacks. If the data showed a varying gray value distribution over one or more data axes, we corrected the histogram along the respective dimension to ensure a homogeneous gray value distribution throughout the entire dataset. As an example, gradients in the HeLa-2 dataset were corrected along the x-and z-axes. Before feeding the data into the CebraEM workflow, we normalized the data by histogram stretching converting the 2-percentile to 0 and the 98percentile to 255 and subsequent conversion to an 8-bit integer data type. The quantiles for normalization were generally confined to regions containing the sample, thus excluding the effect of differing amounts of empty resin within each slice.
The masks were obtained by a rough thresholding-based segmentation performed in Microscopy Image Browser  (40). As in most cases for cellular volume SEM datasets, a significant amount of the data contains empty resin or non-imaged background, the masks used for data normalization were also used to limit computation within the subsequent CebraEM workflow to the areas containing the target cell(s). In the case of the Platynereis-5 dataset, the mask covering ciliated cells was obtained from a previously published cell segmentation and classification  (10). CebraNET architecture and training. The CebraNET is set up based on the 3D U-Net architecture  (23). We set up a concatenation of three 3D U-Net modules consecutively where the last convolution layer of each module forms the input to the subsequent U-Net module.
Each of the three U-Net modules has its sigmoid output layer with binary cross entropy loss. All the U-Net modules are trained as one big network such that the three output layers are treated as three independent output channels. The training was performed with a single 256 x 256 x 64 pixels subset of the HeLa-1 dataset which was manually selected and contained multiple types of intracellular membranes (originating, for example, from mitochondria, the ER, or endosomes) (Fig. 2a ). This ground truth data were annotated for the outer organelle membranes using Microscopy Image Browser, by annotating the present organelles and defining the object boundaries as outer membranes (this saved time as opposed to the annotation of the membranes themselves). To avoid passing contradictory information to the network (e.