dataesr/scientific-paragraphs-categorization huggingface dataset:
Consequently, no input or output masks were used for cleft detection during the prediction of the EM volume. Still, surprising and counterintuitive defects can still arise that require careful testing and dataset augmentation. As an interesting example of a rare failure mode, we mention that folds in the imaging substrate inside a blood vessel lumen (interior) after expansion during alignment were sometimes falsely detected as synapses. To handle this case, we added corrected examples of these errors to the training set and also included a filter to exclude them from the final graph, since the partner assignment step reliably predicts them to be "autapses" of the large blood vessel segments. However, this task's imbalance also represents a challenge in training. Human annotators often miss synapses due to errors of inattention related to this imbalance  (Plaza et al.
Leveraging ground truth of other datasets. A larger training set generally leads to higher accuracy, but ground truth is laborious to generate. To save labor, we reused 943 μm 3 of ground truth from pilot datasets of the MICrONS project (N. L. Turner et al. 2020; Casey M. Schneider-Mizell et al. , 2020; Buchanan et al. 2021; Sven Dorkenwald et al. 2019). A boundary detector pretrained on this old ground truth turned out to generalize well to the new dataset. We further improved accuracy by continuing to train the convolutional net after adding 425 μm 3 of ground truth from the new dataset. These datasets varied in voxel resolution and image quality. For comparison, the recent Drosophila hemibrain reconstruction  (Scheffer et al. 2020 ) relied on 432 μm 3 of manual ground truth and 31,063 μm 3 of semi-automated ground truth to train segmentation models.
Many groups have tried semi-automated creation of ground truth, i. e. , correcting boundaries of an automated segmentation  (Scheffer et al. 2020). The starting automated segmentation tends to be inaccurate in the most difficult and tricky locations, which is where we most need accurate training data to improve the performance of the boundary detector. In our early experiments, we found it more straightforward to annotate such tricky locations from scratch, so we ended up sticking with purely manual annotation. Handling diverse cellular structures. It is relatively straightforward to increase the amount of training data. More nontrivial is the challenge of "covering the problem space with the training data"  (Smith and Topin 2016). The dataset contains diverse cellular structures, and these should be thoroughly sampled by the training data (Figs.
Therefore we found it infeasible to annotate real image defects present in the dataset to create consistent ground truth with certainty. Instead we simulated the defects by distorting clean images and their annotations. Relative to annotating real image defects, this strategy allowed us to intensify the frequency and severity of each kind of defect during simulation to improve the robustness of convolutional nets. This simulation-based approach to training data augmentation previously proven effective for EM images  (Lee et al. 2017; Funke et al. 2017). The training set for the boundary detector included simulated misalignments, as was the case for a pilot dataset  (Sven Dorkenwald et al. 2019; N. L. Turner et al. 2020). It was particularly important to simulate misalignments that co-occurred with missing data, which were common in the aligned stack.
Distributed computation Our petascale reconstruction pipeline makes extensive use of distributed and cloud computing. We processed the dataset using Google Cloud, but our pipeline can also run on other commercial clouds with little or no modification. Because the section images are so large, alignment operations, such as generating and combining transformations, are computed for overlapping patches and blended together. The computations for the patches are distributed over many GPUs. There are minimal task dependencies. The implementation is fairly complex, and will be described elsewhere. Boundary detection, semantic labeling, and synaptic cleft detection model inference were implemented using Chunkflow (J. Wu et al. 2021 ). This software package was developed previously for our terascale pipeline, and extended with only minor changes for the petascale dataset.
Chunkflow divides the input image into overlapping chunks, distributes the processing of the chunks by the convolutional net across multiple workers, and recombines the results of processing into a single output image (Fig. 4a ). Chunkflow uses Docker containers, cloud-based task queues, and Kubernetes for deployment in the cloud. This deployment is fault tolerant, robustly handling task failure through a visibility timeout mechanism. Such failures are inevitable when processing a petascale dataset, which is divided into hundreds of thousands of chunks processed by thousands of cloud GPUs running for several days (J. Wu et al. 2021 ). The distributed computation was simple because there were no task dependencies for different chunks. The agglomeration step of cell segmentation (Fig. 1b ) is in principle a global operation, as the mean affinity criterion summarizes information between two segments across the boundary map.
We used Apache Airflow to manage workflow orchestration. The workflow was described by DAGs (directed acyclic graphs). The segmentation tasks were nodes in these DAGs, with the edges describing the dependencies among the tasks. The auxiliary operations that managed the cloud instance groups were also represented by nodes in the DAGs, the clusters were ramped up when the corresponding segmentation tasks were ready to run, and scaled down when the tasks were finished. Agglomeration required several weeks, sometimes using over 50,000 cloud CPUs at a time. Synapse segmentation and synaptic partner assignment (Fig. 1b ) were implemented using Synaptor. This package includes a distributed implementation of connected components that begins using chunkwise processing (Fig. 4a ), and uses map-reduce processing steps to perform merging across chunks (Fig.
Data access and visualization Each step of the reconstruction pipeline described above (Fig. 1 ) produces several intermediate results. These intermediate results are generally images. We adopted the Precomputed format of Neuroglancer, Google's WebGL-based viewer for volumetric data (Maitin-Shepard 2019), to view these images throughout processing, and this transparency turned out to be indispensable for debugging during pipeline development. Views of problematic locations in image stacks and their effects on the reconstruction could be exchanged effortlessly as URLs by members of the development team. We also built the Igneous package to generate and manage Precomputed data layers. Igneous can create a multiscale image pyramid by fast downsampling of a large image, as well as generate meshes and skeletons from the cell segmentations in Precomputed format.
These features collectively allowed us to diagnose failure modes in the segmentation performance in a way that would be infeasible by viewing the full-resolution segmentation image alone. Igneous generates meshes using marching cubes  (Lorensen and Cline 1987)  and performs mesh simplification using quadric error metrics  (Garland and Heckbert 1997; Hoppe 1999). It performs skeletonization by running Kimimaro, an algorithm based on TEASAR  (Sato et al. 2000; Bitter, Kaufman, and Sato 2001 ) that we further refined. Igneous also handles many chunks within the large EM volumes, so it includes a task queuing mechanism inspired by ChunkFlow to ensure robustness. Pipeline operations also communicated with each other by reading from and writing to cloud storage using CloudVolume, a serverless Python client  (Silversmith 2021).
Reading and writing were typically parallel and multithreaded to maximize throughput, and various options for lossless image compression were used. Of note, fpzip  (Lindstrom and Isenburg 2006)  was used to make the storage of the three-channel 3D floating point affinity maps generated by boundary detection practical. Our development environment also makes public access to the results relatively straightforward. With a few careful considerations of permissions and data management, the public is able to use the same intuitive tools we've used for their own access and exploration. Additionally, Igneous and CloudVolume are designed to avoid dependency on vendor specific features where this is feasible to ensure they will be useful for the foreseeable future.
Discussion We have described a computational pipeline for reconstructing neural circuits in petascale EM datasets. The pipeline incorporates a new approach to image alignment, which was required for dealing with the high frequency of cracks and folds in the EM dataset. Most of the remaining misalignments are quite small, but misalignments are still associated with a significant fraction of segmentation errors. We optimistically predict that the frequency and size of cracks and folds will fall in future datasets, which would lead to reduction of segmentation errors associated with misalignments. According to a contrary view, serial section EM is fundamentally handicapped, and one should instead employ block face EM, a competing 3D EM technique for which alignment challenges are minimal  (Lee et al.
Methods for automated error detection and correction would be helpful for improving segmentation accuracy. Research on such methods has demonstrated promising results  (Zung et al. 2017; Rolnick et al. 2017 ), but has not yet been deployed at scale in our pipeline. EM-based reconstruction of an entire mouse brain has been proposed as a transformative project for neuroscience  (Abbott et al. 2020 ). This would be an exascale dataset, roughly 1000✕ larger than the present one. We expect that most of our pipeline operations should scale to an entire mouse brain, because the tasks for the image chunks have little or no dependencies. The primary exception is hierarchical agglomeration, which proceeds by recursive octree merging (Fig. 4b ). The last step of the recursion at the top of the octree requires a shared memory machine with RAM and CPUs that scale roughly as the square of the linear size of the EM volume.
Alignment at 1024 nm & 64 nm Using the method outlined in  (Buniatyan et al. 2020) , we trained a convolutional network to produce an embedding for each section to improve accuracy during patch matching. We embedded the 1024 ✕ 1024 ✕ 40 nm 3 sections after masking voxels identified as resin to zero. We computed displacement fields between neighboring embedded sections (up to five previous sections) by cross-correlating a co-located 256 x 256 px source patch with a 384 x 384 px target patch at each vertex in a 96 px Cartesian grid. We combined these pairwise displacement fields to generate a displacement field for each section. Using methods similar to those outlined in  (Mitchell et al. 2019) , we trained a convolutional network through self-supervision on the entire rough aligned dataset to predict a displacement field between a pair of 1024 ✕ 1024 ✕ 40 nm 3 sections after embedding as described above, and also took as input the most recently predicted displacement field and crack and fold masks.
We augmented the data with transposing image pairs, as well as translating one image relative to the other up to 4 px at this resolution. We used this network to compute displacement fields between all neighboring sections (up to five previous sections). At inference time, the predicted displacement fields were further refined by using gradient descent to optimize the similarity between the embeddings that were the input to the network. We then combined these pairwise displacement fields by ascending section index to generate a new displacement field for each section. Similar to the above procedure, we also trained a convolutional network to produce displacement fields on 64 ✕ 64 ✕ 40 nm 3 sections. These images were not embedded in advance. Specifically, the network was trained on a subset of the rough aligned dataset to predict a displacement field between a pair of 64 ✕ 64 ✕ 40 nm 3 sections, transforming the highest-level encodings with the most recently predicted displacement field, and with cracks and folds masks used to adjust the similarity and smoothness penalties in the loss function.
The subset consisted of a small initial group of 10 manually chosen samples that contained fold and crack patterns deemed to be particularly difficult, as well as an extended set of 1200 randomly chosen samples within the rough aligned dataset. Each sample covered 5 consecutive sections, allowing for up to 20 permutations per sample. We used this network to compute displacement fields between all neighboring sections. We then created composite sections of neighbors if more immediate neighbors were deemed to have displacements larger than 8 px relative to previous neighbors (evaluated with cross-correlation), or if they contained zero-value pixels. At inference time, the predicted displacement fields were further refined by using gradient descent to optimize the similarity between the lowest-level embeddings produced by the network.
We created an intermediary composite dataset of the aligned partial sections that we used for reconstruction: for each aligned partial section, we created an image that sets each pixel to the value of the first non-zero pixel in the ordered set of sections from the current section to the highest-indexed partial section. We then aligned subvolume 35 in descending order of section index, using the first section in this composite dataset as the initial section. When aligning subvolume 35, we applied a bandpass filter for knife chatter artifacts oriented at 87° to the sections before creating composite images of neighbors.
Segmentation masks To avoid significant segmentation artifacts, we created an input mask for regions of the data remaining misalignments or without useful image content. This mask sets the input to boundary detection to zero in these regions. We also generated an output mask that set the output of boundary detection to zero in other regions. We created these masks by evaluating locations in the aligned dataset which were still misaligned, or did not contain tissue. We produced misalignment maps between each section and its neighbors, up to five sections apart, for both the 256 ✕ 256 ✕ 40 nm 3 aligned dataset, as well as 1024 ✕ 1024 ✕ 40 nm 3 aligned dataset. To compute a misalignment map for a pair of images, we smoothed a thresholded correlation of 8 ✕ 8 px patches that have been transformed and high-passed filtered in the frequency domain.
Boundary Detection Using the method outlined in  (Lee et al. 2017) , we trained a convolutional network to estimate inter-voxel affinities that represent the potential for neuronal boundaries between adjacent image voxels. The network was pretrained using manually annotated ground truth from previously collected cortical datasets (N. L. Turner et al. 2020; Sven Dorkenwald et al. 2019; Buchanan et al. 2021; Casey M. Schneider-Mizell et al. , 2020) , then finetuned on a mix of ground truth collected from this dataset, as well as the previous cortical datasets. We made modifications in network architecture, auxiliary tasks, and data augmentation during training as follows.
Training details Dataset augmentation included (1) flip & rotation, (2) brightness & contrast perturbation, (3) warping, (4) misalignment, (5) missing section, (6) out-of-focus section, (7) misalignment + missing section, (8) lost section, (9) lost + missing section, and (10) slip interpolation. See the dataset augmentation section below. Misalignment augmentation used three different severities of misalignment: (1) small misalignment with maximum displacement of 10 voxels in x/y-dimension, (2) medium misalignment 𝑑 𝑚𝑎𝑥 with of 30 voxels in x/y-dimension, and (3) large misalignment with of 50 voxels in 𝑑 𝑚𝑎𝑥 𝑑 𝑚𝑎𝑥 x/y-dimension. For simulated misalignments, we chose "step" type misalignments with and 𝑝 = 0. 7 "slip" type with  (Lee et al. 2017 ). 𝑝 = 0. 3 We trained our nets on four (pretraining) or eight (finetuning) NVIDIA Titan X Pascal GPUs using synchronous gradient update.
Second, 3 ✕ 3 ✕ 3 convolution filters were used everywhere except for the input and output convolution layers, where 5 ✕ 5 ✕ 1 filters were used. Third, we used upsampling by nearest neighbor interpolation. Fourth, we used Instance normalization  (Ulyanov, Vedaldi, and Lempitsky 2016)  instead of Batch Normalization  (Ioffe and Szegedy 2015). Fifth, we added an extra convolutional layer with filter size of 1 ✕ 1 ✕ 1 at the output layer. Finally, the input patch size to the neural network was fixed to 224 ✕ 224 ✕ 18. The output patch size for each output volume was the same size as the input patch. We used a binary cross entropy loss on each output volume to train the network. We augmented dataset samples using (1) flip & rotate, (2) brightness & contrast, (3) misalignment, (4) missing section, and (5) misalignment + missing section techniques described in the Dataset augmentation section.
Synapse detection and synaptic partner assignment We selected an initial set of 15 training volumes from initial aligned subregions of the full dataset at 8 ✕ 8 ✕ 40 nm 3 resolution, and had human annotators label synaptic clefts within each volume. These volumes were selected to sample failure modes of other datasets -myelin, blood vessels, nuclear membranes, and organelles within somata. We added a region of "padding" to each dataset in order to give human annotators sufficient context to resolve ambiguities without checking the source images. The labeled portions of each volume constituted 227. 6 μm 3 of image data with 135 synaptic clefts. We trained a convolutional network to predict whether a given voxel within each dataset was "synaptic" or "non-synaptic" in the sense that it was labeled to be part of a synaptic cleft.
We also generated output for other regions of the dataset to identify any notable failure modes, and created 10 extra smaller volumes to increase training set representation of synapses onto soma among other errors. We finally added one training volume to decrease network predictions within the interior of blood vessels. This process quadrupled the size of the training set to 914. 0 μm 3 of image data, and 561 total clefts. After finalizing the ground truth, we trained a separate network to produce output on the larger volume. For this training, we used a manual learning rate procedure, decreasing the learning rate by a factor of 10 at apparent convergence of the smoothed validation set error curve. We started training with a learning rate at 1 ✕ 10 -4 , and decreased it 5 times. As before, training used a binary cross entropy loss function.
Unlike the data evaluation experiments, the training data was augmented using (1) flip & rotate, (2) missing section, (3) warping, (4) brightness and contrast, and (5) misalignment techniques described in the Dataset augmentation section below. Misalignment augmentation used one severity of misalignment with of 10 voxels in x/y-dimension, and we chose "step" type misalignments with 𝑑 𝑚𝑎𝑥 and "slip" type with. We stopped training after 1. 23 million training updates. 𝑝 = 0. 7 𝑝 = 0. 3 We then trained a separate network to perform synaptic partner assignment by producing the voxels of the synaptic partners given the synaptic cleft as an attentional signal (Nicholas L. We made discrete predictions of synaptic clefts by performing connected components throughout the large volume. We performed connected components in 2048 ✕ 2048 ✕ 256 voxel chunks, and then ran additional processing to merge the predictions from these chunks together.
Dataset augmentation Each convolutional network was trained using a set of dataset augmentations to improve the network's robustness to imaging artifacts and other variability in the image data. Below we detail the set of data augmentations used across each network described above. • Flip & rotation ○ Each input patch to the convolutional net is randomly flipped in x and y dimensions, and then rotated by multiples of 90 degrees. The random combinations of rotation by 90° and horizontal/vertical flips are only applied in the xy-plane due to the voxel anisotropy of the EM images. This procedure yields eight 2D configurations. Additional flip in z-dimension doubles the number of configurations, resulting in a total of 16 configurations. This augmentation is commonly used in image processing applications.
• Warping ○ We randomly warped the input images by combining the following linear transformations: (1) continuous rotation, (2) shear, (3) twist, (4) scale, and (5) perspective stretch. Again we adapted the code from ElektroNN (Sven  Dorkenwald et al. 2017 ). • Misalignment  (Lee et al. 2017)  ○ We simulated misalignments with varying severity specified by a maximum displacement. The displacement in each dimension is randomly sampled from an integer-valued 𝑑 𝑚𝑎𝑥 discrete uniform distribution in [0, ]. For each simulated misalignment, we randomly 𝑑 𝑚𝑎𝑥 chose the type of misalignment ("slip" vs. "step", see the Segmentation Masks section) with a predefined probability. • Missing section  (Lee et al. 2017)  ○ We simulated (1) two consecutive missing sections, either partial or full (randomly chosen), and (2) up to seven independent single missing sections, either partial or full (randomly chosen).
• Lost + missing section ○ We additionally introduced the lost + missing section augmentation. Here we replace three consecutive image sections in the input with a single missing section, and leave only the middle of the three corresponding target sections in the ground truth segmentation, so that a net can learn to interpolate the missing section as smoothly as possible in the challenging co-occurrence of missing and lost sections. To avoid any problematic situation, we modified our data augmentation logic such that simulation of (1) misalignment, (2) missing section, (3) misalignment + missing section, (4) lost section, and (5) lost + missing section are all mutually exclusive. • Slip interpolation  (Lee et al. 2021)  ○ The dominant type of misalignment errors in the current dataset are so-called "slip" misalignments, in which sections at z-1 and z+1 are well-aligned to each other whereas the section at z is misaligned with respect to both the sections above and below.
Acknowledgements The authors thank  David Markowitz , the  IARPA MICrONS Program Manager , who coordinated this work during all three phases of the  MICrONS program. We thank  IARPA program  managers  Jacob Vogelstein  and  David Markowitz  for conceiving of and advocating for the MICrONS program. We thank  Wade Leonard  and  Jennifer Wang ,  IARPA SETAs  for their assistance. The work was supported by the  Intelligence Advanced Research Projects Activity (IARPA)  via  Department of Interior/ Interior Business Center (DoI/IBC)  contract numbers  D16PC00003 ,  D16PC00004 , and  D16PC0005. The  U. S. Government  is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright annotation thereon. HSS also acknowledges support from  NIH/NINDS   U19 NS104648 ,  NIH/NEI   R01 EY027036 ,  NIH/NIMH   U01 MH114824 ,  NIH/NIMH   U01 MH117072   NIH/NINDS   R01 NS104926 ,  NIH/NIMH   RF1 MH117815 ,  NIH/NIMH   RF1 MH123400  and the  Mathers Foundation , as well as assistance from  Google ,  Amazon , and  Intel.