TITLE: Multiplex IHC Image Processing V0.2
AUTHORS: Shamilene Sivagnanam, Courtney Betts, Lisa Coussens

[DESCRIPTION]
Understanding immune complexity within the tumor microenvironment provides valuable insight to tumor-immune composition, spatial interactions, and the immune system's reaction to therapy allowing us to better understand response and resistance at the interface of the tumor and host. 
Multiplex IHC allows for the simultaneous detection of multiple Ab targets in the same tissue section using published methods previously described1,2,3.This protocol contains detailed instructions on how to prepare files and run the image processing pipeline for mIHC SVS images in a GUI or command line, making it accessible to biologist and labs without computational specialty. 



Tsujikawa, Takahiro, et al. "Quantitative multiplex immunohistochemistry reveals myeloid-inflamed tumor-immune complexity associated with poor prognosis."Cell reports19.1 (2017): 203-217
Banik, Grace, et al. "High-dimensional multiplexed immunohistochemical characterization of immune contexture in human cancers."Methods in enzymology635 (2020): 1-20.
Liudahl SM, Betts CB, Sivagnanam S, et al. "Leukocyte Heterogeneity in Pancreatic Ductal Adenocarcinoma: Phenotypic and Spatial Features Associated with Clinical Outcome." Cancer Discov. 2021;11(8):2014-2031. doi:10.1158/2159-8290.CD-20-0841

[BEFORE_START]
Make sure you have access to the following commercial softwares:


Leica, Aperio ImageScope (free, Windows only)
 


Matlab R2018b or R2019b (license required)
 
Additional toolboxes required: Computer Vision Toolbox


[GUIDELINES]
This protocol is specifically for chromogenic multiplexIHC image processing using images scanned with Leica, Aperio ScanScope in the .SVS format. 

[STEPS]
SECTION: File prep and Folder Structure
1. 
FOLDER AND FILE NAMING CONVENTION

Make sure folder and file naming convention is consistent throughout the entire study. Filenaming convention should be used when saving images during acquisition. 

 


SECTION: File prep and Folder Structure
2. 
SET UP FOLDER STRUCTURE

The pipeline is optimal for batches of slides that run from a 'Parent' folder, that is the study or batch directory.

   

The study folder structure should be organized as the cohort/study/batch of slides as the parent folder, with each slide within the study containing all marker files with no duplicate scans of markers. 

There should be an initial hematoxylin, a second hematoxylin after antigen retrieval, and a third hematoxylin at the end of the panel - see mIHC wet lab protocol for details: 

Below is an example of the folder structure of a study called HTAN1 that contains 1 slide (H342253), 17 marker images, and 3 hematoxylin stained images. 

 



SECTION: Get Code and Required Software
3. 
DOWNLOAD SOFTWARE AND CODE

Download or clone the MultiplexIHC_Code.zip file that contains all the functions and code for image processing. Save and unzip this folder somewhere other than your Downloads directory.

git@github.com:Sivoyer/MultiplexIHC_Code.git


In addition to this code, you will also need the following software installed:

Registration is performed in Matlab R2018b or higher:
 
Additional toolboxes required: Computer Vision Toolbox


Color Deconvolution and Nuclei Segmentation is performed in FIJI:
 
Additional files required:  
To install this .class file, drag and drop into FIJI bar, then close and restart the program.


Single Cell Quantification is performed in Cell Profiler 3:
 



SECTION: Pipeline Overview
4. 
PIPELINE OVERVIEW

Once file names and folder structure is properly formatted for mIHC image processing, batch analysis can be performed. For every slide, multiple regions can be selected, avoiding regions of tissue deformation, necrosis, or other artifact or trouble tissue regions. 

The image processing workflow utilizes steps commonly found in biological image analysis that have been optimized for brightfield RGB multplexIHC images with AEC chromogen and hematoxylin. Although image processing can be completely automated, it is highly recommended that each step is checked to ensure data is as expected and is of high enough quality to continue processing. The modular steps allow for QC and adjustments at each major step. 

Brief Description of Image Processing Workflow 
Each region is registered to the final stained hematoxylin image from the same area, aligning each image such that major shifts due to repetitive scanning are adjusted for. Color deconvolution separates the AEC conjugated Ab signal from background. Single cell nuclei segmentation identifies cell areas using hematoxylin stain. Each identified cell is measured for mean intensity of every signal extracted marker.


    

SECTION: Region of Interest Selection
5. 
SELECT REGIONS OF INTEREST 

Regions are annotated in Aperio ImageScope, available for Windows only.
 

Annotating Regions on final hematoxylin
Regions of interest are annotated on the last hematoxylin image to ensure that only cells maintained throughout all staining cycles and rounds are evaluated in downstream analysis. 

Tissue areas that are lost, deformed, folded, or of poor quality should be excluded from analysis. 
 


SECTION: Workflow Options
6.  
Pipeline Run Options

This can be run using the software interfaces (code free), or by terminal. Instructions for both methods are listed below:
SECTION: Image Registration
7. 
IMAGE REGISTRATION

How registration works:

Image registration is performed in Matlab. Each marker image is registered to the last cycle target hematoxylin image. Keypoints are identified on the blue channel of the hematoxylin image and the blue channel of the AEC marker image.
If not enough matching keypoints are found, other single RGB channels are checked for more keypoints, and the channels with the highest strongest matching indexed keypoints are used. A similarity based geometric transform is generated and applied to each channel of the marker image and merged. The registered RGB image is saved as a TIFF inside an automatically generated 'Registered_Regions' folder created in the slide/region folder.

  
 
Replace <path/to/matlab> with the path to your matlab application. For macOS, it is likely: /Applications/MATLAB_R2018b.app/bin/matlab

Replace <path/to/matlab/code-dir> with the path to the location where you saved the MultiplexIHC_RegistrationCode directory.

Replace <path/to/Parent-dir> with the path to your Parent directory containing all slides to be registered.

Example:

 



SECTION: Image Registration Quality Control
8. 
REGISTRATION QC

When registration is complete, you will see a Registered_Regions directory created inside each slide folder with an ROI folder for each region box annotation. Each ROIxx folder contains all marker images for that region with reg_ prepended to the filename.

A new folder called 'Registration_Check' will be created in the Parent directory that contains low resolution images of all registered regions for every slide in the batch. Each low res image will have 'regck_' prepended to the filename. This image set can be used to quickly check the images for gross misregistration or images that were were false registered (passed but should have failed). Regions or stains that did not adequately register well enough for quantification due to tissue deformation or loss are removed from further analysis.



  

If registration failed for a marker or region, you will see a 'Redo_ROI' folder inside the slide folder containing the failed marker region and the hematoxylin region that failed registration. 

Images that failed registration can be excluded from further downstream analysis, or manually registered. See Troubleshooting guide ### for how to register images - 'Manual Registration' XXX.

SECTION: Signal Extraction
9. 
AEC COLOR DECONVOLUTION

Color deconvolution separates the AEC signal from the tissue background to quantify signal.

AEC signal is extracted using an RGB to CMYK conversion which utilizes a maximum gray component replacement to subtract the lowest gray level from all channels. AEC chromogen signal is separated into the ‘Y’ and 'M' channels and the 8-bit image channels are added together. Pixel results greater than 255 are set to 255. The middle 90% of pixels are rescaled to 0-255 (Pham et al. 2007). 

  
 
EXAMPLE:

 

The script runs on the Parent directory and creates a directory in each slide called 'Processed' with ROI folders inside; same folder structure as 'Registered_Regions'.  Each generated image will have 'V_' prepended to the filename. 

  
 

SECTION: Image Registration
7. 
IMAGE REGISTRATION

How registration works:

Image registration is performed in Matlab. Each marker image is registered to the last cycle target hematoxylin image. Keypoints are identified on the blue channel of the hematoxylin image and the blue channel of the AEC marker image.
If not enough matching keypoints are found, other single RGB channels are checked for more keypoints, and the channels with the highest strongest matching indexed keypoints are used. A similarity based geometric transform is generated and applied to each channel of the marker image, then merged back into an RGB image. The registered RGB image is saved as a TIFF inside an automatically generated 'Registered_Regions' folder created in the slide/region folder.

  
Registration Steps:

1. Open Matlab and set working path by selecting the 'choose folder' icon and set the path to the location you saved the MultiplexIHC_Code directory. On the left panel called 'Current Folder' you will see the files listed in gray. 

2. Right click on the MultplexIHC_Code directory in the 'Current Folder' panel and select 'Add to Path' > 'Selected Folders and Subfolders'. They will turn from gray to black indicating they are available. 

  
 

SECTION: Image Registration QC
8. 
REGISTRATION QC

When registration is complete, you will see a Registered_Regions directory created inside each slide folder with an ROI folder for each region box annotation. Each ROIxx folder contains all marker images for that region with reg_ prepended to the filename.

A new folder called 'Registration_Check' will be created in the Parent directory that contains low resolution images of all registered regions for every slide in the batch. Each low res image will have 'regck_' prepended to the filename. This image set can be used to quickly check the images for gross mis-registration or images that were were false registered (passed but should have failed). Regions or stains that did not adequately register well enough for quantification due to tissue deformation or loss are removed from further analysis.

  

If registration failed for a marker or region, you will see a 'Redo_ROI' folder inside the slide folder containing the failed marker region and the hematoxylin region that failed registration. 

Images that failed registration can be excluded from further downstream analysis, or manually registered by selecting matching keypoints from both images. See Troubleshooting guide ### for how to register images - 'Manual Registration' XXX.

SECTION: Signal Extraction
9. 
AEC COLOR DECONVOLUTION

Color deconvolution separates the AEC signal from the tissue background to quantify signal.

AEC signal is extracted using an RGB to CMYK conversion which utilizes a maximum gray component replacement to subtract the lowest gray level from all channels. AEC chromogen signal is separated into the ‘Y’ and 'M' channels and the 8-bit image channels are added together. Pixel results greater than 255 are set to 255. The middle 90% of pixels are rescaled to 0-255 (Pham et al. 2007). 

    
 


The script runs on the Parent directory and creates a directory in each slide called 'Processed' with ROI folders inside; same folder structure as 'Registered_Regions'.  Each generated image will have 'V_' prepended to the filename. 

   





SECTION: Nuclei Segmentation
10. 
NUCLEI SEGMENTATION

Segmentation is performed using the StarDist 2D_versatile_fluo algorithm implemented in FIJI python:
stardist/stardist: StarDist - Object Detection with Star-convex Shapes (github.com)
(Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, and Gene Myers.
Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy.
The IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, Colorado, March 2020) 

 
Segmentation can be performed on any single channel nuclear stain, preferably from the end of the staining panel to ensure only cells maintained throughout all staining cycles are evaluated downstream. If using a nuclear stain early in the panel, tissue and cells lost during the staining process can result in false negatives. We have optimized an antibody cocktail of H3+Nuclear Antigen to robustly identify single cell nuclei and this is recommended for segmentation in this pipeline.

 


SECTION: Single Cell Measurements
11. 
SINGLE CELL MEASUREMENTS

Single cell measurements are performed in Cell Profiler. Although there are many methods to measure mean intensity values for masked objects, this workflow utilizes Cell Profiler due to the output compatibility with FCS Express Image Cytometry for single cell classification. Measurements necessary for single cell gating are mean intensity for each marker, object area, object index, and object centroid X,Y location coordinates. 

There are specific nomenclature that is required in order for the Cell Profiler output file to work properly in FCS Express Image Cytometry.

See the attached document on how to use the template Cell Profiler project, which can be found in the MultiplexIHC_Code directory. 

  




SECTION: Visualization stack (optional)
12. CONVERT TO OME.TIFF

The 'V_' prepended images are your registered, signal extracted, single channel 8-bit final image. Each set of images within an ROI can be converted into a single ome.tif using bioformats to create Level 2 data.


SECTION: Single Cell Measurements
11.  SINGLE CELL MEASUREMENTS

Single cell measurements are performed in Cell Profiler. Although there are many methods to measure mean intensity values for masked objects, this workflow utilizes Cell Profiler due to the output compatibility with FCS Express Image Cytometry for single cell classification. Measurements necessary for single cell gating are mean intensity for each marker, object area, object index, and object centroid X,Y location coordinates. 

There are specific nomenclature that is required in order for the cell profiler output file to work properly in FCS Express Image Cytometry.

See the attached document on how to use the template Cell Profiler project, which can be found in the MultiplexIHC_Code directory. 

  
 
Example of running 50 image sets from the file Batch_data.h5 located in the project folder, Cohort1, on Linux:
 

 
Example of running 50 image sets from the file Batch_data.h5 located in the project folder, Cohort1 on MacOS (change CellProfiler version to your version):
 


This command submits the batch file to CellProfiler and specifies that CellProfiler run in a batch mode without its user interface to process the pipeline. This run can be modified by
using additional options to CellProfiler that specify the following:

-p <Default_Output_Folder_path>/Batch_data.h5: The location of the batch file, where <Default\_Output\_Folder\_path> is the output folder path as seen by the cluster
     computer.
 
-c: Run “headless”, i.e., without
     the GUI
 
-r: Run the pipeline specified on
     startup, which is contained in the batch file.
 
-b: Do not build extensions, since
     by this point, they should already be built.
 
-f <first_image_set_number>: Start processing with the image set specified,
     <first_image_set_number>
 
-l <last_image_set_number>: Finish processing with the image set specified,
     <last_image_set_number>

SECTION: Segmentation
10. NUCLEI SEGMENTATION

Segmentation is performed using the StarDist 2D_versatile_fluo algorithm implemented in FIJI python:
stardist/stardist: StarDist - Object Detection with Star-convex Shapes (github.com)
(Martin Weigert, Uwe Schmidt, Robert Haase, Ko Sugawara, and Gene Myers.
Star-convex Polyhedra for 3D Object Detection and Segmentation in Microscopy.
The IEEE Winter Conference on Applications of Computer Vision (WACV), Snowmass Village, Colorado, March 2020) 

Segmentation can be performed on any single channel nuclear stain, preferably from the end of the staining panel to ensure only cells maintained throughout all staining cycles are evaluated downstream. If using a nuclear stain early in the panel, tissue and cells lost during the staining process can result in false negatives. We have optimized an antibody cocktail of H3+Nuclear Antigen to robustly identify single cell nuclei and this is recommended for segmentation in this pipeline.

 
BatchStardistSegmentation_H3NUCA.py
The script runs on the Parent directory and creates a labeled image for each ROI inside the Processed folder with a prefix, 'label_', based on the nuclei image named with 'V_H3NUCA_*'. This can be modified to identify any nuclei image name.