# Artificial Intelligence and generative data augmentation for automated chromosomal aberration detection in cytogenetic FISH imaging

Quentin Tallon, Emmanuel Moebel, Juan Martinez, Eric Gregoire, Pascale Fernandez, Delphine Dugue, Rose Deboucha, Laure Bobyk, Marco Valente, Gaetan Gruel, et al.

► To cite this version:

Quentin Tallon, Emmanuel Moebel, Juan Martinez, Eric Gregoire, Pascale Fernandez, et al. Artificial Intelligence and generative data augmentation for automated chromosomal aberration detection in cytogenetic FISH imaging. ERRS 2024 - 48h European Radiation Research Society Meeting, Sep 2024, Averiro, Portugal. pp.1-24. hal-04902922

HAL Id: hal-04902922

https://inria.hal.science/hal-04902922v1

Submitted on 21 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
<figure><img src="image_2.png" /></figure>

RÉPUBLIQUE
FRANÇAISE

Liberté
Egalité
Fraternité

IRSN

INSTITUT DE RADIOPROTECTION
ET DE SÛRETÉ NUCLÉAIRE

ARTIFICIAL INTELLIGENCE AND GENERATIVE DATA
AUGMENTATION FOR AUTOMATED CHROMOSOMAL
ABERRATION DETECTION IN CYTOGENETIC FISH IMAGING

QuentinTALLON¹,²

with E. Moebel², J. Martinez¹, E. Gregoire¹,
P. Fernandez¹, D. Dugue¹, R. Deboucha¹,
L. Bobyk³, M. Valente³, G. Gruel¹,
C. Kervrann², and M.A. Benadjaoud¹

¹ LRAcc, PSE-SANTE/SERAMED IRSN

² SAIRPICO, INRIA Rennes

³ IRBA

Inria

MEMBRE DE
ETSON

INSTITUT DE RADIOPROTECTION
ET DE SÛRETÉ NUCLÉAIRE
## Biological Dosimetry Context

<figure><img src="image_3.png" /></figure>

Accidental or malicious exposure of people to ionizing radiation

<figure><img src="image_4.png" /></figure>

Blood collection and placement in cell culture (48h)

<figure><img src="image_5.png" /></figure>

Acquisition of metaphases

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

GIEMSA

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>

FISH

Inverse regression of the
calibration curve for **dose**
estimation and its uncertainties

### Limitations

* **Manual** counting

* **Cognitive** fatigue

* Not enough qualified people

* Complex task: noise, change of acquisition equipment, chromosome polymorphism

Counting aberrations
(most time-consuming step)
## FISH Metaphase Imagery

<figure><img src="image_12.png" /></figure>

Facilitate the **counting** of chromosomal aberrations by biologists

Two tasks:

- **Localization** of chromosomes of interest

- **Classification** of localized chromosomes

We focus on **fluorescent** chromosome detection

Aberrations visible thanks to **color co-localization**

AI in computer vision through deep learning: need of **images** and **corresponding annotations**
Aberration Counting Sheets

Manuscript

<figure><img src="image_13.png" /></figure>

Digitized

<figure><img src="image_14.png" /></figure>

The LRAcc annotation is a **count annotation**:

* one counting sheet corresponds to one microscope slide

* one line corresponds to one metaphase

* we don't know the **spatial position** of aberrations

Work on **digitization** of handwritten counting sheets: Juan Martinez, Eric Grégoire and Rose Déboucha (LRAcc)
Need for a spatial annotation: encoding the analysis methodology

Aberrations visible thanks to the **co-locations of colors** so
spatial coordinates are essential

<figure><img src="image_15.png" /></figure>

Encoding of
expertise

<figure><img src="image_16.png" /></figure>

Transmitting information to AI
models

<figure><img src="image_17.png" /></figure>

<figure><img src="image_18.png" /></figure>
Effort to homogenize FISH images

Homogenization of data induces less confusion of vision models

This corresponds to the **counting methodology** of experts in operational situations

In particular, the correction of **excess fluorescence** (recalibration of the DAPI quantiles)

<figure><img src="image_19.png" /><figcaption>Original quantiles</figcaption></figure>

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

<figure><img src="image_23.png" /><figcaption>Corrected quantiles</figcaption></figure>
Texture Correction Overview

BEFORE

<figure><img src="image_24.png" /></figure>

16602074L1-A.087.PNG

<figure><img src="image_25.png" /></figure>

16602074L1-A.088.PNG

<figure><img src="image_26.png" /></figure>

16602074L1-A.089.PNG

<figure><img src="image_27.png" /></figure>

16602074L1-A.090.PNG

<figure><img src="image_28.png" /></figure>

16602074L1-A.092.PNG

<figure><img src="image_29.png" /></figure>

16602074L1-A.093.PNG

<figure><img src="image_30.png" /></figure>

16602074L1-A.094.PNG

<figure><img src="image_31.png" /></figure>

16602074L1-A.095.PNG

<figure><img src="image_32.png" /></figure>

16602074L1-A.100.PNG

<figure><img src="image_33.png" /></figure>

16602074L1-A.101.PNG

<figure><img src="image_34.png" /></figure>

16602074L1-A.110.PNG

<figure><img src="image_35.png" /></figure>

16602074L1-A.111.PNG

AFTER

<figure><img src="image_36.png" /></figure>

16602078L1-A.087.PNG

<figure><img src="image_37.png" /></figure>

16602078L1-A.088.PNG

<figure><img src="image_38.png" /></figure>

16602078L1-A.089.PNG

<figure><img src="image_39.png" /></figure>

16602078L1-A.090.PNG

<figure><img src="image_40.png" /></figure>

16602078L1-A.092.PNG

<figure><img src="image_41.png" /></figure>

16602078L1-A.093.PNG

<figure><img src="image_42.png" /></figure>

16602078L1-A.094.PNG

<figure><img src="image_43.png" /></figure>

16602078L1-A.095.PNG

<figure><img src="image_44.png" /></figure>

16602078L1-A.100.PNG

<figure><img src="image_45.png" /></figure>

16602078L1-A.101.PNG

<figure><img src="image_46.png" /></figure>

16602078L1-A.110.PNG

<figure><img src="image_47.png" /></figure>

16602078L1-A.111.PNG
## Creation of the final database: FISH Audit

Adding spatial information to our data using bounding boxes

<figure><img src="image_48.png" /></figure>
<table><tr><td>Contexte</td><td>Données</td><td>Métaphase</td><td>Chromosome</td><td>Génératif</td></tr></table>

Overview of the databases

<table><thead><tr><th></th><th>Images</th><th>Normal<br>chromosomes</th><th>Translocated<br>chromosomes</th><th>Aberrant<br>chromosomes</th></tr></thead><tbody><tr><td>DATASET 1<br>semi-supervised</td><td>357</td><td>2308</td><td>275 (11%)</td><td>N / A</td></tr><tr><td>DATASET 2<br>manual</td><td>452</td><td>2561</td><td>154 (5%)</td><td>328 (11%)</td></tr><tr><td>DATASET 3<br>manual<br>FISH Audit</td><td>3476</td><td>18103</td><td>3123 (17%)</td><td>5962 (33%)</td></tr></tbody></table>

DATASET 1:

- annotations obtained in a semi-supervised manner

- only the *translocations* have been spatially annotated

DATASET 2 & 3 (FISH Audit):

- manual annotations

- all the *aberrations* are spatially annotated

Imbalanced classes: *translocations* are *underrepresented*
## First detections at the metaphase scale

The two detection tasks are to be solved **simultaneously**:

- Localization of chromosomes of interest

- Classification of localized chromosomes

Nomenclature of the chromosome classes:

- Class "color": normal fluorescent chromosomes

- Class "aberration": aberrant chromosomes

**Supervised** object detection method: images and spatial annotations

- 80% for **training and validation**

- 20% for **testing**

Evaluation of **model performances** on test data

<figure><img src="image_49.png" /></figure>
State of the art in object detection: Faster-RCNN

Faster-RCNN uses region proposal network (RPN) to speed up the object detection process

<figure><img src="image_50.png" /></figure>
## Results obtained with Faster-RCNN

$$ \text{Precision} = \frac{tp}{tp + fp} $$

$$ \text{Recall} = \frac{tp}{tp + fn} $$

$$ F = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} $$

<table><thead><tr><th>Class</th><th>Metric</th><th>DATASET 1<br>semi-supervised</th><th>DATASET 2<br>manual</th><th>DATASET 3<br>manual<br>FISH Audit</th></tr></thead><tbody><tr><td rowspan="2">Color</td><td>Precision</td><td>0.76</td><td>0.95</td><td>0.86</td></tr><tr><td>Recall</td><td>0.77</td><td>0.96</td><td>0.94</td></tr><tr><td rowspan="3">Aberration</td><td><strong>F1</strong></td><td><strong>0.76</strong></td><td><strong>0.95</strong></td><td><strong>0.90</strong></td></tr><tr><td>Precision</td><td>0.66</td><td>0.50</td><td>0.76</td></tr><tr><td>Recall</td><td>0.23</td><td>0.33</td><td>0.67</td></tr><tr><td></td><td><strong>F1</strong></td><td><strong>0.35</strong></td><td><strong>0.40</strong></td><td><strong>0.71</strong></td></tr></tbody></table>

Similar scores for the Faster-RCNN on GIEMSA Audit (A. Deschemps PhD 2020-2023)

Voting system with heatmaps and model aggregation is possible

Strengthen decision making through the color class (6 colors in a normal metaphase)
# Task separation to circumvent aberration rarity

## Step 1

Localization of chromosomes of interest

<figure><img src="image_51.png" /></figure>

<figure><img src="image_52.png" /></figure>

## Step 2

Classification by convolutional networks such as ResNet

<figure><img src="image_53.png" /></figure>
## Good segmentation allows work at the chromosome scale

Cell nuclei segmentation model: *Cellpose*

* Based on the *probability of presence* of a cell

* Instance segmentation: each chromosome is distinguished from the others

Principle of *fine tuning*: adaptation of a model to new data

<figure><img src="image_54.png" /></figure>
## Chromosome scale classification performance

ResNet: *residual convolutional* neural network

<figure><img src="image_55.png" /></figure>

<table><thead><tr><th>Class</th><th>Metric</th><th>ResNet (Chromosome)</th><th>Faster-RCNN (Metaphase)</th></tr></thead><tbody><tr><td rowspan="3">Color</td><td>Precision</td><td>0.97</td><td>0.86</td></tr><tr><td>Recall</td><td>0.97</td><td>0.94</td></tr><tr><td><strong>F1</strong></td><td><strong>0.97</strong></td><td><strong>0.90</strong></td></tr><tr><td rowspan="3">Aberration</td><td>Precision</td><td>0.76</td><td>0.76</td></tr><tr><td>Recall</td><td>0.70</td><td>0.67</td></tr><tr><td><strong>F1</strong></td><td><strong>0.73</strong></td><td><strong>0.71</strong></td></tr></tbody></table>

Comparable performance between the metaphase scale and that of the chromosome which validates data preprocessing (segmentation + homogenization)
Exploring generative diffusion models

Reminder of the major challenges:

- Imbalanced classes

- Little data compared to the needs of the models

The generation of *synthetic translocations* is
of major interest

Two possible cases of generation:

- Noise → Image

- Image → Image: possibility of DAPI input to generate fluorescence

<figure><img src="image_56.png" /></figure>

Denoising Diffusion Probabilistic Models

**Jonathan Ho**
UC Berkeley
jonathanho@berkeley.edu

**Ajay Jain**
UC Berkeley
ajayj@berkeley.edu

**Pieter Abbeel**
UC Berkeley
pabbeel@cs.berkeley.edu
# Brownian Bridge Diffusion Model (BBDM)

## Model Image → Image:

- Allows you to use the large amount of *unannotated DAPI*

- Reduces the generation to the *fluorescence*

## Forward diffusion:

- Iterative corruption (Markov chain) of the original image $Z_0$ with gaussian noise

## Backward/Reverse diffusion:

- Iterative de-corruption with neural network model (U-Net architecture)

- For a noisy image at the step $Z_t$, find the noise added in step $t-1$

<figure><img src="image_57.png" /></figure>

**Brownian Bridge Processes**
Diffusion Model Test Pipeline

Generate chromosomes

<figure><img src="image_58.png" /></figure>

Add to FISH Audit data

<figure><img src="image_59.png" /></figure>

Compare ResNet scores

<figure><img src="image_60.png" /></figure>

Two generative databases:

<table><thead><tr><td>Data generated</td><td>Number of images</td></tr></thead><tbody><tr><td>BASE 1: DAPI of fluorescent chromosomes</td><td>1154</td></tr><tr><td>BASE 2: DAPI of non-fluorescent chromosomes</td><td>10179</td></tr></tbody></table>

Examples of generations

<figure><img src="image_61.png" /></figure>
Using ResNet intermediate layers for filtering

An image is associated to a feature vector, *penultimate ResNet layer*

UMAP dimension reduction (an image is represented by a point)

<figure><img src="image_62.png" /></figure>

<figure><img src="image_63.png" /></figure>

Translocations are concentrated
## UMAP Projection of Generative Databases

<figure><img src="image_64.png" /><figcaption>BASE 2: from DAPI of non-fluorescent chromosomes</figcaption></figure>

<figure><img src="image_65.png" /></figure>

<table><thead><tr><td>Data</td><td>BASE 1</td><td>BASE 2</td></tr></thead><tbody><tr><td>Number of images filtered by UMAP ResNet</td><td>820</td><td>7518</td></tr></tbody></table>
<table><tr><td>Contexte</td><td>Données</td><td>Métaphase</td><td>Chromosome</td><td>Génératif</td></tr></table>

## F1 scores of the aberration class using the generated data

<table><caption>BASE 1: from DAPI of fluorescent chromosomes</caption><thead><tr><td>Case study</td><td>Score</td><td>Cardinal</td></tr></thead><tbody><tr><td>Real data</td><td>73%</td><td>N = 1021</td></tr><tr><td>Generated Data</td><td>70%</td><td>N = 820</td></tr><tr><td>Real + Generated Data</td><td>80%</td><td>N = 1841</td></tr></tbody></table>

<table><caption>BASE 2: from DAPI of non-fluorescent chromosomes</caption><thead><tr><td>Case study</td><td>Score</td><td>Cardinal</td></tr></thead><tbody><tr><td>Real data</td><td>73%</td><td>N = 1021</td></tr><tr><td>Generated Data</td><td>68%</td><td>N = 7518</td></tr><tr><td>Real + Generated Data</td><td>76%</td><td>N = 8539</td></tr></tbody></table>

We must generate translocations from morphologically close DAPI

Allows us to reuse DAPI from fluorescent chromosomes of normal metaphases
## Dosimetric Validation

* Construction of an **automatic counting tool** chromosomal translocations from standardized FISH images.

* Need to demonstrate implementation **operational work**: characterization **counting statistics** of the AI model and validation by **dose estimation**.

* Calibration curve modeling with base rate dependent on **demographic variables**

<figure><img src="image_66.png" /></figure>
Contexte

Données

Métaphase

Chromosome

Génératif

Perspectives

mFISH imagery greatly improves the quality of the data and thus the quality of the detection

<figure><img src="image_67.png" /></figure>
IRSN

INSTITUT DE RADIOPROTECTION
ET DE SÛRETÉ NUCLEAIRE

THANK YOU FOR YOUR
ATTENTION!

MEMBRE DE

ETSON

INSTITUT DE RADIOPROTECTION
ET DE SÛRETÉ NUCLEAIRE