dataesr/scientific-paragraphs-categorization huggingface dataset:
For both types of dataset, we find that fewer than ten annotated nuclei are sufficient to initiate a virtuous cycle of training, prediction and proofreading, which efficiently yields cell tracks and validated cell lineages in highly dynamic tissues. Interactive cycles of manual annotation, deep learning and proofreading on ELEPHANT reduce the time required to detect and validate nuclei (Figure  2a ). On the CE1 dataset, a complete cell lineage was built over 195 timepoints, from scratch, using ELEPHANT's semi-automated workflow (Figure  2c ). The detection model was trained incrementally starting from sparse annotations (four nuclei) on the first timepoint. On this dataset, linking could be performed using the nearest neighbor algorithm (without flow modeling) and manual proofreading. In this way, we were able to annotate in less than 8 hours a total of 23,829 nuclei (across 195 timepoints), of which ~2% were manually annotated (483 nuclei) and the remaining nuclei were collected by validating predictions of the deep-learning model.
Although ELEPHANT works efficiently without prior training, cell tracking can be accelerated by starting from models trained on image data with similar characteristics. To illustrate this, we used nuclear annotations in a separate dataset, CE2, to train a model for detection, which was then applied to CE1. This pre-trained model allowed us to detect nuclei in CE1 much more rapidly and effortlessly than with an untrained model (Figure  2a , blue versus orange curves). For benchmarking, the detection and linkage models trained with the annotations from the CE1 and CE2 lineage trees were then tested on unseen datasets with similar characteristics (without proofreading), as part of the Cell Tracking Challenge  6, 14. In this test, our models with assistance of flow-based interpolation (see Methods) outperformed state-of-the-art tracking algorithms  16, 17  in detection (DET) and tracking (TRA) metrics (Figure  2b ).
The PH dataset presents greater challenges for image analysis, such as larger variations in the shape, intensity, and distribution of nuclei, lower temporal resolution, and more noise (Supplementary Figure  5 ). ELEPHANT has allowed us to grapple with these issues by supporting the continued training of the models through visual feedback from the user (annotation of missed nuclei, validation and rejection of predictions). Using ELEPHANT, we annotated and validated over 260,000 nuclei in this dataset, across 504 timepoints spanning 168 hours of imaging. We observed that the conventional nearest neighbor approach was inadequate for linking in the PH dataset, resulting in many errors in the lineage trees (Figure  2d ). This is likely due to the lower temporal resolution in this dataset (20 minutes in PH, versus 1-2 minutes in CE) and the fact that daughter nuclei often show large displacements at the end of mitosis.
We trained optical flow using 1,162 validated links collected from 10 timepoints (including 18 links for 9 cell divisions). These sparse annotations were sufficient to generate 3D optical flow predictions for the entire dataset (Supplementary Video 2), which significantly improved the linking performance (Figure  2d , Supplementary Table  2 ): the number of false positive and false negative links decreased by ~57% (from 2,093 to 905) and ~32% (from 1,991 to 1349), respectively, among a total of 259,071 links. By applying ELEPHANT's human-in-the-loop semi-automated workflow, we succeeded in reconstructing 109 complete and fully-validated cell lineage trees encompassing the duration of leg regeneration in Parhyale, each lineage spanning a period of ~1 week (504 timepoints, Supplementary Figure  6 ).
Using analysis and visualization modules implemented in Mastodon and ELEPHANT, we could capture the distribution of cell divisions across time and space (Figure  2e ) and produce a fate map of the regenerating leg of Parhyale (Figure  2f ). This analysis, which would have required several months of manual annotation, was achieved in ~1 month of interactive cell tracking in ELEPHANT, without prior training. Applying the best performing models to new data could improve tracking efficiency even further.
Image datasets The PH dataset (dataset li13) was obtained by imaging a regenerating T4 leg of the crustacean Parhyale hawaiensis, based on the method described by  15  (Supplementary Video 1). The imaging was carried out on a transgenic animal carrying the Mi(3xP3>DsRed; PhHS>H2B-mRFPRuby) construct  18  , in which nuclear-localised mRFPRuby fluorescent protein is expressed in all cells following heat-shock. The leg was amputated at the distal end of the carpus. Following the amputation, continuous live imaging over a period of 1 week was performed on a Zeiss LSM 800 confocal microscope equipped with a Plan-Apochromat 20x/0. 8 M27 objective (Zeiss 420650-9901-000), in a temperature control chamber set to 26˚C. Heat-shocks (45 minutes at 37˚C) were applied 24 hours prior to the amputation, and 65 and 138 hours post-amputation.
Every 20 minutes we recorded a stack of 11 optical sections, with a z step of 2. 48 microns. Voxel size (in xyz) was 0. 31 x 0. 31 x 2. 48 microns. The CE1 and CE2 datasets were from  13  , obtained via the Cell Tracking Challenge 14 (datasets Fluo-N3DH-CE). ELEPHANT platform architecture ELEPHANT implements a client-server architecture (Supplementary Figure  1 ), which can be set up on the same computer or on multiple connected computers. This architecture brings flexibility: allowing the client to run Mastodon (implemented in Java) while the deep learning module is implemented separately using Python, and releasing the client computer from the requirements of high GPU needed to implement deep learning. The client side is implemented by extending Mastodon, a framework for cell tracking built upon the SciJava ecosystem (https://scijava.
org/) and is available as a Fiji 5 plugin. Combining the BigDataViewer  19  with an efficient memory access strategy (https://github. com/mastodonsc/mastodon/blob/master/doc/trackmate-graph. pdf), Mastodon enables fast and responsive user interaction even for very large datasets. ELEPHANT leverages the functionalities provided by Mastodon, including the functions for manual annotation of nuclei, and extends them by implementing modules for deep learning-based algorithms. The server side is built using an integrated system of a deep learning library (PyTorch 20 ), tools for tensor computing and image processing (Numpy 21 , Scipy  22  , Scikit Image  23  ) and web technologies (Nginx, uWSGI, Flask). The client and the server communicate by Hypertext Transfer Protocol (HTTP) and JavaScript Object Notation (JSON).
To reduce the amount of data exchanged between the client and the server, the image data is duplicated and stored in an appropriate format on each side. An in-memory data structure (Redis) is used to organize the priorities of the HTTP requests sent by the client. A message queue (RabbitMQ) is used to notify the client that the model is updated during training. The client software is available as a standalone Java executable packaged with Mastodon and other dependencies (https://github. com/elephant-track/elephant-client). The server environment is provided as a Docker container to ensure easy and reproducible deployment (https://github. com/elephanttrack/elephant-server).
Computer setup and specifications In this study, we set up the client and the server on the same desktop computer (Dell Alienware Aurora R6) with the following specifications: Intel Core i7-8700K CPU @ 3. 70GHz, Ubuntu 18. 04, 4x16 GB DDR4 2666 MHz RAM, NVIDIA GeForce GTX 1080 Ti 11 GB GDDR5X (used for deep learning), NVIDIA GeForce GTX 1650 4 GB GDDR5, 256 GB SSD and 2 TB HDD. System requirements for the client and the server are summarized in the user manual (https://elephant-track. github. io/).
Dataset preparation Images were loaded in the BigDataViewer (BDV,  19  ) format on the client software. The CE1 and CE2 datasets were converted to the BDV format using the BigDataViewer Fiji plugin (https://imagej. net/BigDataViewer)without any preprocessing. Because the PH dataset showed non-negligible variations in intensity during long-term imaging, the original 16-bit images were intensity normalized per timepoint before conversion to the BDV format; intensity values were re-scaled so that the minimum and maximum values at each timepoint become 0 and 65535, respectively. The PH dataset also showed 3D drifts due to heat-shocks. The xy drifts were corrected using an extended version of image alignment tool  24  working as an ImageJ  25  plugin, where the maximum intensity projection images were used to estimate the xy displacements, subsequently applied to the whole image stack (https://github.
com/elephant-track/align-slices3d), and the z drifts were corrected manually by visual inspection using Fiji. On the server, images, annotation labels and outputs were stored in the Zarr format, allowing fast read/write access to subsets of image data using chunk arrays. At the beginning of the analysis, these data were prepared using a custom Python script that converts the original image data from HDF5 to Zarr and creates empty Zarr files for storing annotation labels and outputs (https://github. com/elephant-track/elephant-server). Algorithm for detection Detection of nuclei relies on three components: (i) a U-Net CNN that outputs probability maps for nucleus center, nucleus periphery, and background, (ii) a post-processing workflow that extracts nucleus center voxels from the probability maps, (iii) a module that reconstructs nuclei instances as ellipsoids.
For each group normalization layer, the number of groups is set as the smallest value between 32 and the number of output channels, and the weights and biases are respectively initialized to one and zero. When starting to train from scratch, the CNN is trained using the cropped out 3D volumes from the original image prior to training with annotations. In this prior training phase, a loss function  "#$%# is used that penalizes the addition of the following two mean absolute differences (MADs): (i) nucleus center probabilities  $ and the [0, 1] normalized intensity of the original image  $ , (ii) background probabilities  $ and the [0, 1] normalized intensity of the intensity-inverted image 1 - $ , where stands for the voxel index of an input volume with voxels. The prior training is performed on three cropped out 3D volumes generated from the 4D datasets, where the timepoints are randomly picked, and the volumes are randomly cropped with random scaling in the range (0.
8, 1. 2). The training is iterated for three epochs with decreasing learning rates (0. 01, 0. 001, and 0. 0001, in this order) with the Adam optimizer  29. The prior training can be completed in ~20 seconds for each dataset. Training with sparse annotations is performed in the following steps. First, the client application extracts the timepoint, 3D coordinates and covariances representing ellipsoids of all the annotated nuclei in the specified time range. Subsequently, these data, combined with user-specified parameters for training, are embedded in JSON and sent to the server in an HTTP request. On the server side, training labels are generated from the received information by rendering nucleus center, nucleus periphery, background and unlabeled voxels with distinct values. The background labels are generated either by explicit manual annotation or intensity thresholding, resulting in the label images as shown in Figure  1b.
To render ellipsoids in the anisotropic dimension, we extended the draw module in the scikit-image library  23  (https://github. com/elephant-track/elephant-server). Training of the CNN is performed using the image volumes as input and the generated labels as target with a loss function that consists of three terms: (i) a class-weighted negative log-likelihood (NLL) loss, (ii) a term computed as 1 minus the dice coefficient for the nucleus center voxels, and (iii) a term that penalizes the roughness of the nucleus center areas. We used the empirically-defined class weights for the NLL loss: nucleus center = 10, nucleus periphery = 10, background = 1; the unlabelled voxels are ignored. The first two terms accept different weights for the true annotations (i. e. true positive and true negative) and the false annotations (i.
Training of the CNN is performed on the image volumes generated from the 4D datasets, where the volumes are randomly cropped with/without random scaling, random contrast and random rotation, which are specified at runtime. There are two modes for training: (i) an interactive mode that trains a model incrementally, as the annotations are updated, and (ii) a batch mode that trains a model with a fixed set of annotations. In the interactive training mode, sparse annotations in a given timepoint are used to generate crops of image and label volumes, with which training is performed using the Adam optimizer with a learning rate specified by the user. In the batch training mode, a set of crops of image and label volumes per timepoint is generated each iteration, with which training is performed for a number of epochs specified by the user (ranging from 1 to 1,000) using the Adam optimizer with the specified learning rates.
First, on the client application, for each annotated link, which connects the source and target nuclei, the following information gets extracted: the timepoint, the backward displacements in each of the three dimensions, and the properties of the target nucleus (3D coordinates and covariances). Subsequently, these data, combined with parameters for training, are embedded in JSON and sent to the server in an HTTP request. On the server side, flow labels are generated from the received information by rendering backward displacements for each target nucleus in each of three dimensions, where the displacements are scaled to fit the range (-1, 1). In this study, we used fixed scaling factors (1/80, 1/80, 1/10) for each dimension, but they can be customized to the target dataset. Foreground masks are generated at the same time to ignore unlabelled voxels during loss calculation.
5. Using these components and the empirically-defined weights for each loss term ( ), we defined the as below. , , , , where and. The training is performed on the image volumes generated from the 4D datasets, where the sets of two consecutive images and corresponding flow labels are randomly cropped with/without random scaling and random rotation, which are specified at runtime. The training is performed for a fixed number of epochs using the Adam optimizer and with learning rates specified by the user, generating a set of images and labels for each timepoint in each epoch. The CNN outputs are rescaled to the original physical scale and used to calculate the estimated coordinate of each nucleus center at the previous timepoint. Let stands for a subset of voxel index of a nucleus and for its center coordinate.
Detection and tracking in the CE datasets On the CE1 and CE2 datasets, training of detection and flow models was performed with volumes of 384 x 384 x 16 voxels that were generated by preprocessing with random scaling in the range (0. 5, 2) and random cropping. For training of a detection model, preprocessing with random contrast in the range (0. 5, 1) was also applied. In the label generation step, the center ratio was set to 0. 3 and the background threshold was set to 0. 1 and 1 (i. e. all voxels without manual annotations are background), for the interactive mode and the batch training mode, respectively. In the interactive training of detection models, 10 labelled cropped out volumes were generated per iteration, with which training was performed using the Adam optimizer with a learning rate between 5x10 -5 and 5x10 -6.
In the postprocessing of the prediction for detection, a threshold for the nucleus center probabilities were set to 0. 3, and and were set to 1 and 3, respectively. In the nearest-neighbor linking with/without flow prediction, was set to 5 µm and was set to 3. In the results submitted to the CTC organizer (Figure  2b ), the linking was performed by the nearest-neighbor linking with flow support and an optional interpolation module, where was set to 5. Detection and tracking in the PH dataset On the PH dataset, training of detection and flow models was performed with volumes of 384 x 384 x 12 voxels generated by preprocessing with random rotation in the range of +/-180 degrees and random cropping. For training a detection model, preprocessing with random contrast in the range (0. 5, 1) was also applied.
Analysis of CE and PH datasets On the CE1 and CE2 datasets, the detection and link annotations were made starting from timepoint 0 and proceeding forward until timepoints 194 (CE1) and 189 (CE2), respectively. In the CE1 dataset, the detection was made from scratch, based on manual annotation and incremental training, and the linking was performed by the nearest neighbor algorithm without flow prediction. After completing annotation from timepoint 0 to 194 on the CE1 dataset, the detection and flow models were trained by the batch mode with the fully-labeled annotations. In the CE2 dataset, the detection was performed in a similar way as for CE1, by extending the model trained with CE1, and the linking was performed by the nearest neighbor algorithm with flow support using the pre-trained model followed by proofreading.
Incremental training of the detection model was performed when there were annotations from nuclei that were not properly predicted. On the PH dataset, the annotations were made by iterating the semi-automated workflow. In general, the nuclei with high signal-to-noise ratio (SNR) were annotated early, while the nuclei with low SNR were annotated in a later phase. The detection model was updated frequently to fit the characteristics of each region and timepoint being annotated, while the flow model was updated less frequently. The CE1 dataset was used to evaluate the speed of detection and validation (Figure  2a ). All workflows started at timepoint 0 and proceeded forward in time, adding and/or validating all the nuclei found in each timepoint. To evaluate the manual workflow, we annotated nuclei using hotkeys that facilitate the annotation of a given nucleus at successive timepoints.
To evaluate the ELEPHANT from scratch workflow, we performed prediction with the latest model, followed by proofreading, including add, modify or delete operations, and incremental training. At each timepoint, the model was updated with the new annotations added manually or by proofreading. To evaluate the ELEPHANT pre-trained workflow, we performed predictions with a model trained on the CE2 dataset, followed by proofreading without additional training. The numbers of validated nuclei associated with time were counted from the log data. We measured the counts over 30 minutes after the start of each workflow and plotted them in Figure  2a. To compare the linking performances (Figure  2d ), we trained the flow model with 1,162 validated links, including 18 links corresponding to 9 cell divisions, from 108 lineage trees collected between timepoints 150 and 159.
Evaluation of cell tracking performance We submitted our results and executable software to the Cell Tracking Challenge organizers, who evaluated our algorithm's performance, validated its reproducibility using the executable software that we submitted, and provided us with the scores. The details of the detection accuracy (DET), tracking accuracy (TRA), and segmentation accuracy (SEG) metrics can be found in the original paper  36  and the website (http://celltrackingchallenge. net/evaluationmethodology/). Briefly, the DET score evaluates how many split, delete and add operations are required to achieve the ground truth starting from the predicted nuclei, reflecting the accuracy of detection; the TRA score evaluates how many split, delete and add operations for nuclei, and delete, add and alter the semantics operations for links are required to reconstruct the ground truth lineage trees from the predicted lineage trees, reflecting the accuracy of linking; the SEG score evaluates the overlap of the detected ellipsoids with fully segmented nuclei, reflecting the precision of nucleus segmentation.
Bottom right: The trained model generates voxel-wise probability maps for background (magenta), nucleus center (blue), or nucleus periphery (green). Bottom left: The user validates or rejects the predictions; predicted nuclei are shown in green, predicted and validated nuclei in cyan. c, Linking workflow, illustrated on the CE1 dataset. Top left: The user annotates links by connecting detected nuclei in successive timepoints; annotated/validated nuclei and links are shown in cyan, nonvalidated ones in green. Top right: The flow model is trained with optical flow labels coming from annotated nuclei with links (voxels indicated in the label mask), which consist of displacements in X, Y and Z; greyscale values indicate displacements along a given axis, annotated nuclei with link labels are outlined in red.
Right panels: Corresponding lineage trees. d, Comparison of tracking results obtained on the PH dataset, using the nearest neighbor algorithm (NN) with and without optical flow prediction (left panels); linking errors are highlighted in red on the correct lineage tree. The panels on the right focus on the nuclear division that is marked by a dashed line rectangle. Without optical flow prediction, the dividing nuclei (in magenta) are linked incorrectly. e, Spatial and temporal distribution of dividing cells in the regenerating leg of Parhyale over a 1-week time course (PH dataset), showing that cell proliferation is concentrated at the distal part of the regenerating leg stump and peaks after a period of.
Acknowledgements We are grateful to  Anna Kreshuk  and  Constantin Pape  for training in machine learning, to  Jean-Yves Tinevez  and  Tobias Pietzsch  for support in developing ELEPHANT as a Mastodon plugin, to the  NEUBIAS community  for feedback on the software, to the Cell Tracking Challenge organizers for support in our submission to the challenge, and to  Sebastien Tosi  for extensive feedback on the manuscript. We also thank  Jan Funke ,  Carsten Wolff ,  Martin Weigert ,  Jean-Yves Tinevez ,  Philipp Keller ,  Irepan Salvador-Martínez ,  Severine Urdy , and  Mathilde Paris  for comments on the manuscript. This research was supported by the  European Research Council , under the  European Union   Horizon 2020 programme , grant ERC-2015-AdG # 694918 ; CC was supported by a doctoral fellowship from  Boehringer Ingelheim Fonds.
Code availability The source code for the ELEPHANT client is available at https://github. com/elephanttrack/elephant-client, for the ELEPHANT server at https://github. com/elephanttrack/elephant-server, and for the Align Slices 3D+t extension ImageJ plugin at https://github. com/elephant-track/align-slices3d. The user manual for ELEPHANT is available at https://elephant-track. github. io/.
The table shows a summary of linking performances tested on the PH dataset, on a total number of 259,071 links (including 688 links on cell divisions). Incremental training was performed by transferring the training parameters from the model pre-trained with the CE datasets. Linking performance on dividing cells is scored separately. Training from scratch Incremental training from pre-trained model Nearest neighbour all 905 959 2,093 False positive cell division 11 16 16 all 1,349 1,329 1,991 False negative cell division 232 306 327 Training time 31 h 4 min -.