TITLE: Generating Nuclei Segmentations for vEM datasets using Cellpose 
AUTHORS: Emma Avetissian, Alyson Petruncio, Jeff Rhoades, Aubrey Weigel, CellMap Project Team

[DESCRIPTION]
This protocol details the process of generating nucleus segmentations in FIB-SEM (Focused Ion Beam Scanning Electron Microscopy) image stacks of cells and tissues using Cellpose. These segmentations are crucial for gaining biological insights and training deep learning models on cell and organelle segmentation. The protocol covers the application of Cellpose for fine-tuning 2D models specifically for nuclei segmentation. Once fine-tuned, the model is used to generate 3D predictions for the entire dataset. These segmentations can be further refined through manual approaches, utilized for visualization, or employed in training other machine learning models.  

[STEPS]
SECTION: Introduction:
1. The purpose of this protocol is to provide comprehensive instructions on how to create nuclei segmentations in FIB-SEM datasets on OpenOrganelle.org using Cellpose. 

Cellpose is a generalist algorithm developed by Stringer et al., (2020) to segment cellular structures. Cellpose also provides software for manual segmentation and model training. In this protocol we discuss the use of Cellpose v3.0.9.
 
OpenOrganelle.org is an online platform that provides access to high-resolution 3D electron microscopy images of cellular structures for scientific research and education.

In this protocol you can expect to find: 
OpenOrganelle Overview 
Using Fiji to save files in the cloud locally as 2D tiffs
Cellpose in 2D and Model Training 
Cellpose in 3D and Predictions 
Final Touches 
SECTION: OpenOrganelle Overview
2. OpenOrganelle (Heinrich et al, 2021) is a collection of focused ion beam scanning electron microscopy (FIB-SEM) datasets curated by the Janelia CellMap Project Team. Each dataset on OpenOrganelle includes important background information about the sample, protocol, contributors, and more. The “VIEW” button opens a new tab to view the chosen dataset in Neuroglancer. Each dataset is also accompanied by a Fiji link which copies the location of the dataset in the S3 bucket (see Figure 1). This link was used to read data into Fiji and save as 2D slices for Cellpose. 

 

 

 

SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
3. Once the Fiji link from OpenOrganelle has been copied, it can be pasted into Fiji. 

 

 

SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
4. Head to: File → Import → HDF5/N5/Zarr/... 
SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
5. Paste the link into the search bar. 

SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
7. Navigate through the file container by clicking the dropdown, e.g.  “em”.
Choose resolution, e.g., s0 (original), s1 (2x downsample), s3 (4x downsample), etc.  

 


SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
9. Once the dataset is open in Fiji, it can be saved as a sequence of 2D slices for reading into Cellpose.
From the open dataset window go to: 
File → Save As → Image Sequence → Save into a new folder
See Figure 3. 
 

 

SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
8. Click OK. See Figure 2.

SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
6. Click Detect Datasets
SECTION: Using Fiji to save files in the cloud locally as 2D tiffs
10. Before closing the image in Fiji, save a second copy of the dataset as a 3D tiff. This will be used for generating 3D predictions in a later step in Cellpose. If the image is closed before creating the 3D tiff, you can simply drag the folder containing the image sequence into the main Fiji bar to open it. To save it as a 3D tiff: 
File → Save As → Tiff 
See Figure 4. 

 

 

 

SECTION: Cellpose in 2D and Model Training
11. To begin using Cellpose on the dataset, open the program (see Appendix). More detailed instructions for general Cellpose use can be found on https://cellpose.readthedocs.io/en/latest/gui.html. Within Cellpose go to: 
File → Load Image → Select a slice for segmentation 
See Figure 5.

 

 

 


SECTION: Cellpose in 2D and Model Training
12. Annotate the nuclei as accurately as possible.

At first, it may be beneficial to use an existing model to make annotating easier. Out of the box, the cyto2_cp3 model tends to give promising results for nuclei in FIB-SEM volumes of tissue. If the model does not catch anything, try annotating a couple slices manually and training a model. Continue to use and improve the model by periodically adding more training slices. 

 

 Refer to Figure 6 for what a completed slice might look like. 

 

 

SECTION: Cellpose in 2D and Model Training
13. Once the annotation is complete, save the slice along with the masks.  
File → Save masks and image (as *_seg.npy) 

This file will automatically be saved into the folder where the rest of the 2D slices can be found. 

 

SECTION: Cellpose in 2D and Model Training
14. To train a model on annotated slices, go to: 
Models → Train new model with image+masks in folder 
SECTION: Cellpose in 2D and Model Training
15. To run this model on a new 2D slice
SECTION: Cellpose in 2D and Model Training
14.1. Set the initial model to cyto2_cp3

SECTION: Cellpose in 2D and Model Training
14.2. Adjust n_epochs to change the number of epochs (we generally went with 1000)
SECTION: Cellpose in 2D and Model Training
14.3. Name the model for organizational purposes. Click “OK” and wait for the training to finish.

 

 

SECTION: Cellpose in 2D and Model Training
15.1. Open the blank 2D slice and locate the section labeled “Other models” on the left-hand side of the program.
SECTION: Cellpose in 2D and Model Training
15.2. Open the dropdown for custom models, select the new model, and click run. 
SECTION: Cellpose in 2D and Model Training
15.3. Repeat this process, making use of previous models, annotating additional slices until the model performs satisfactorily on 2D slices.
  

SECTION: Cellpose in 3D and Predictions
16. Cellpose can extend models that were trained exclusively on 2D data to perform predictions in 3D. Once the model demonstrates a satisfactory performance in 2D, we attempt to run a prediction in 3D.

 

SECTION: Cellpose in 3D and Predictions
16.1. Close Cellpose 
SECTION: Cellpose in 3D and Predictions
16.2. Into the terminal enter:

 

This will alert Cellpose to load data as 3D 
SECTION: Cellpose in 3D and Predictions
16.3. Now load in the previously saved 3D volume by: 
File → Load Image → Select the tiff file 
SECTION: Cellpose in 3D and Predictions
17. Once the tiff has loaded, run the custom model as described for 2D in Step 15.

This may take a while (on the order of tens of minutes) and progress can be tracked in the terminal.

The process will be complete when the masks are visible (see Figure 8).

 

 

The terminal will also contain how many masks were found at the end of the run (see Figure 9).  

 

SECTION: Cellpose in 3D and Predictions
18. To save the masks go to: 
File → Save masks as PNG/tiff