Assessing Quality of Tissue Engineered Retinal Pig

Epithelia Using Absorbance Imaging & Artificial Intell

July 1, 2021

Carl Simon
Age-Related Macular Degeneration & 

Retinal Pigment Epithelial Cells

(AMD & RPE)

https://www.cdcionhealth/basicdex.html

• RPE support rods & cones by delivering nutrients from 

the bloodstream & removing waste that the rods and cones generate

• In AMD, RPE stop performing their support functions & 

rods & cones die, resulting in loss of central vision

• AMD is a common cause of vision loss in developed 

countries, affecting 30 to 50 million people worldwide

• No good treatment for 90% of AMD cases (dry AMD)

• Treatment Goal: Manufacture healthy RPE & deliver to 

eye to prevent rod and cone cell death

Jha BS, Bharti K. Regenerating Retinal Pigment Epithelial Cells to Cure Blindness: A Road Towards Personalized Artificial Tissue. Curr Stem Cell Rep (2015) 1:79-91.

Formation of tight 

junctions is key for 

RPE function
80 days 75 days

iPSCs Neuro- ectoderm iRPE progenitors Committed iRPE I

Blood

RPE Manufacturing in Bharti Lab at NIH: Takes 155 Days 

• Draw patient blood, centrifuge, 

retain PBMC fraction

• Isolate CD34+ monocytes using 

magnetic beads

• Generate bank of clinical grade 

induced-pluripotent stem cells (iPSCs) using 4 episomal factors 

AMD Patient-Derived iPSCderived RPE (iRPE)

CD34+ 

Peripheral 

Blood 

Monocytes

P
Kamao et al. Objective Evaluation of the Deg

Pluripotent Stem Cell–Derived RPE. Invest O

Pigmentation Correlates with RPE Maturation

• RPE express melanin to absorb light• Prevents light from entering the back of the eye

• Reduce light scattering in the eye to improve 

vision 

• Protect tissue from exposure to light, reduce 

disease & cancerous lesions

• NOTE: Pigmentation is an indirect measure of function 

• Pigmentation is not part of the mechanism of 

action for treating AMD

• MOA = support of the rods & cones

Degree of Pigmentation

Culture 
Cruz et al. Nature 

Cruz et al. 2018: Phase 1 Clinical Study of an 

Embryonic Stem Cell-Derived RPE Patch in AM

“Appearance” is a key quality attribute 

“RPE cells were assessed using a light microscope 

for pigmentation, cobblestone morphology, health 

and signs of contamination and processed further, 

only if they passed this visual check.”

“On the day of surgery...T

visually through the clear lid 

for integrity, pigmented cell 
Measurement Issue

How to replace “Appearance” qualitative visual inspection tests using 

quality attribute of iRPE patches with quantitative measurem
Calculate Ab

A ൌ െlog

Shutter Open (IMax Shutter Closed (IMin) RPE ( ) ICell) 

• Use brightfield microscope as a 

spectrophotometer

• Each pixel in an image is a quantitative

measure of pigmentation

Quantitative Bright-Field Absorbance Microscopy (Q
Brightfield
Absorbance
The information in the pixels is used to assess RPE quality by 2 appr

1. Deep neural network (DNN): uses image-level spatial patterning 

intensities• Avoids image processing/feature engineering• Creates a “Black box” model• Tissue-level & cell-level

2. Traditional Machine Learning (TML): uses single cell-level metric

intensity, texture• Requires extensive image processing/feature engineering• Provides some biological insights based on important cell metri• Cell-level only

Experimental Overview
0.0

0.20.40.60.81.0

0.0 0.2 0.4 0

Microscope Absorbance

Spectrophotometer A

654C

Compare QBAM with spec

Data Collection Routines: using neutral densit• Data collection automated using custom plugin 

for MicroManager

• Starts with “microscope stability” routine where 

it collects images at different exposure times until 95% confidence interval of each pixel is 0.01 absorbance units

• The best exposure time for each pixel is used 

to generate an “optimized” image

• Takes images at 3 wavelengths by using three 

different filters (red, green, blue)

• This helps to account for scattering, since 

scattering is less dependent on wavelength than is absorbance

QBAM Quality Assurance
QBAM Quality Assurance

Assessing comparability 

absorbance of 3 different n

on 3 different microsco

Stability Metric

Time (min) Spectrophotom

QBAM Abs (mAU) • Capture 3 images and average (I1) • Capture another three images and average (I2)

• the microscope is stable if stability metric is 0 ± 0.0001

Stability metric ൌ െLogଵ଴

Iଵ

Iଶ
Experimental Overview

iPSCs Immature iRPE Seed into Transwell Plates Culture 8 weeks with weekly measurements of: QBAM, TER & VEGF

Segment 

cell borders in QBAM by 

DNN-S

Train 

predict T

from Q

5/6 of 

Convert to 

Absorbance

Images

Patient 

Blood

Determin

single ce

metrics 

using WIP

DNN-F
Experimental Overview

Image by 

fluorescence

Segment 

fluorescence images 

for cell borders using 

Fogbank + handcorrections

Wide variety of 

mature RPE with 

cell borders 

immunostained

for ZO-1

Train DNN-Z to predict ZO-1 

segmentations from ZO-1 

fluorescence images using 

Fogbank+hand-corrected

segmentations as truth

Image by 

QBAM & 

fluorescence

Segment fluorescence images 

for cell borders 

using DNN-Z + 

hand-corrections

Mature iRPE with 

cell borders 

immunostained

for ZO-1

Train DNN-S to p

borders in QBA

DNN-Z+hand-co

segmentations 

Co-register 

QBAM & fluorescence images 

so they perfectly 

overlay

DNN-ZDNN-S

Goal for DNN-Z & DNN-S: Segment single cells in QBAM images• By using QBAM images to predict 

where a human would identify a cell border in ZO-1 immunostainedfluorescence image

iPSCs Immature iRPE Seed into Transwell Plates Culture 8 

weekly me

of: QBAM, 

Patient 

Blood

ZO-1 Cell Border
B

A

H

CC A H

B

C

A

HH C A

B

HC

AA H C

BA

H

CC A H

B

C

A

HH C A

B Blank

C Control

A Aphidicolin

H HPI4

Empty

• 12-well transwell plates• 6 plates x 6 wells/plate = 36 wells• iPSC-derived RPE from healthy donor• 3 treatments (control, aphidicolin, HPI4)• QBAM

• 12 overlapping fields of view per well (4 x 3) per time point • Use three different color filters (red, green, blue)

• TER measurement on all wells and plates 1X/week for 6 weeks• VEGF measured on each well in 2 plates 1X/week

Train on 5 plates...test on 1 plate

B

HC

AA H C

Cell seeding

Start treatments

Experimental Design
Generating Good & Bad RPE: Aphidicolin & HP

Aphidicolin• inducer of RPE maturation • antibiotic that inhibits eukaryotic nuclear DNA 

replication and blocks the cell cycle at early S phase

HPI4• inhibitor of RPE maturation • hedgehog pathway inhibitor-4, HPI4• Hedgehog signaling pathway transmits information to 

embryonic cells required for proper cell differentiation
Mam

cultexhInve

Tight junctions are key for RPE function

Basal side

Apical side

World Precision 

Instruments, LLC

TER 

Tight junctions: multiprotein junctional complexes common in epithelial cell layers that function to prevent passage of solutes & water between cells

biologydictionary.net

1) Trans-Epithelial Resistanc2) Polarized VEGF Secretion 

RPE Functional Attributes:
Fun Facts

Scope• iRPE Manufacturing takes 155 days

• Bharti lab has 2 dedicated staff that only 

manufacture RPE for use by the rest of the lab

• $10K in growth factors required to make a batch of 

RPE

• Cost per patient is unknown (~$1M per patient)• iPSC technology is new, discovered in 2006, only 6 

patients have rec’d them

• 6 in Japan (1 iRPE patch & 5 iRPE suspensions)• 0 in USA

• Data from iRPE from 10 donors• Implemented 53 DNN & ML AI routines to• Data Dissemination (600 GB): 

https://isg.nist.gov/deepzoomweb/data/RPEimplants

Challenges• Reliability: microscope stability, 

background, blank, 3 colors, diff

• Speed: Optimizing speed to min

are out of the incubator

• Big data: 200K images, 1TB, 12

• 4 channels of data: QBAM 

fluorescence

• organizing, moving, annota

• Processing: background subtrac

absorbance images from brightf

• Stitching: 4x3 tiling from each w• 7 summer students participated • Build prototype at NIST & convin• GLP: NIH gave us access to GL
0.1 mm

mAU

Control

Week 1

Aphidicoli

(induces differen

HPI4

(inhibits differentiation)

Week 8

Quantitative Brightfield Absorbance Microscopy (Q
Each data point is a well

RPE Maturation: Mean Absorbance, TER & VEGF R

Weekly QBAM imaging did not impact iRPE maturation
Root Mean Squared Error (RMSE)

Accuracy: 94%, Sensitivity: 1.0, Sp

Mean Absorbance vs. AI: Trans-Epithelial Resistance 

Quality threshold set at “400” 

(could be refined after clinical trial)

DNN-F
Root Mean Squared Error (R

Accuracy: 100%, Sensitivity: 

Mean Absorbance vs. AI: VEGF Ratio

Quality threshold set at “3.0” 

(could be refined after clinical trial)

DNN-F
Truth: DNN-Z segmentation QBAM image

of ZO-1 fluorescence Images

DNN-S segmentation DNN-S overlaid on truth 

Deep Neural Network Prediction of RPE Image Segme

Truth 

DNN-S 

Overlap

Comparative histimage features oby hand or by DN• F-2 = 0.71
“Cobblestone Network”

1 89 2345

6

7

Number of Neighbors
Machine Learning Prediction of TER from WIPP Single Ce

Summ

Reg

AlgorithmDNN-F: Deep LNeural Network MLP: Multi-LayPerceptron PLSR: Principlsquares RegreL-SVM: Linear Vector Machine RR: Ridge RegRF: Random F

R2 = 0.94

False 

Positive

False 

Negative

True 

Positive

Multi-Layer Perceptron 

Predicted TER (Ωꞏcm2)

dashed line represents 

a perfect prediction

Measured TER (Ωꞏcm

2)
Heatmap of the cell image feature importance of all features 

analyzed across top 4 performing machine learning algorithms

Different TML Models Key in on a Similar Set of Cell 

• Intensity 1 = average minimum intensity per cell• Texture 1 = 3rd inverse difference moment at 45°

• Texture 2 = 3rd inverse difference moment at 135°• Shape 1 = Zernike n4- l0 polynomialWhat is the biological significance of these metrics?
Different TML Models Key in on a Similar Set of Cell 

• Intensity 1 = average minimum intensity per cell• Texture 1 = 3rd inverse difference moment at 45°

• Texture 2 = 3rd inverse difference moment at 135°• Shape 1 = Zernike n4- l0 polynomialWhat is the biological significance of these metrics?

250200150100

2345 7 6

Minimum Intensity Per 

Cell (mAU)

Time (Weeks)

Z

4

0

Zernike n4- l0 polynomial

Local heterogeneity is non-homogenous

Low 

Pigmentation
Machine Learning Prediction of VEGF Ratio from WIPP Singl

Summa

Regre

AlgorithmDNN-F: Deep LeaNeural Network RF: Random ForeMLP: Multi-Layer Perceptron L-SVM: Linear SuVector Machine PLSR: Principle Lesquares RegressioRR: Ridge Regres

False 

Positive

False 

Negative

True 

Positive

dashed line represents 

a perfect prediction

Random Forest 

Predicted VEGF Ratio

Measured VEGF Ratio
Results not discussed

• Results confirmed in additional donors... 

• iRPE from 5 albino patients: verify QBAM measurements on iRPE 

diagnosable differences in pigmentation

• iRPE from 2 healthy donors: to verify that QBAM did not impact iR• Clinical-grade iRPE from 3 AMD donors (and 2 or 3 clones from ea

donors): to verify AI predictions from QBAM worked for iRPE from A

• Using AI to predicting donor identity from QBAM images

• FDA requires STR phenotyping of for manufactured autologous ce
Conclusions

• Weekly QBAM imaging did not impact iRPE maturation (non-invasive• Unprocessed QBAM could predict iRPE function: TER & VEGF ratio• DNN of unprocessed QBAM images more accurate than segmentatio• Very important that training set have good & bad samples

• Time: early timepoints less mature, later timepoints more mature• Treatments: HPI4 inhibits RPE maturation, aphidicolin promotes• BALANCE: If you only feed the AI great samples, then it will pred

the test samples are great
Dissemination

• Schaub NJ, Hotaling NA, Manescu P, Padi S, Wan Q, Sharma R, George A, Chalfoun 

Simon Jr CG, Bajcsy P, Bharti K (2020) Deep learning predicts function of live retinal pquantitative microscopy. Journal of Clinical Investigation 130, 1010-1023.

• https://doi.org/10.1172/JCI131187

• QBAM:

• SQuIRE: Micromanager plugin to collect images

• Github: https://github.com/Nicholas-Schaub/SQuIRE

• CARPE: ImageJ plugin fto convert bright-field microscope images into absorbanc

• Github: https://github.com/Nicholas-Schaub/CARPE

• Data: https://isg.nist.gov/deepzoomweb/data/RPEimplants

• iRPE from healthy donors

• Healthy1, live RPE, broadband (232 GB)• Healthy2, live RPE, narrowband (291 GB) 

• iRPE from 3 AMD patients (4 GB): 2 or 3 clones per donor, cells were fixed• iRPE from 5 albino patients (36 GB): Cells were fixed• QBAM and ZO-1 fluorescence images from segmenting routines (4 GB)
Acknowledgements

NIST/ITL

Petre Manescu

Sarala Padi

Aman George

Joe Chalfoun

Mylene Simon

Mohamed Ouladi

Peter Bajcsy

NIST/MML

Nicholas Schaub

Nathan Hotaling

Michael Halter

Anne Plant

John Elliott 

Carl Simon

NIH/NEI

Kapil Bharti

Arvydas Maminishkis

Ruchi Sharma

Qin Wan

Davide Ortolan

Tareq Uddin

Nyusha Lin

Nikhil Etikela

Malika Nimmagadda

Peter Stewart

Helen Zhao

Marissa Coene

Tochi Ugweje

Jun Jeon

Hannah Bush