# From statistical learning to deep learning for improved modeling of the nuisance component in high-contrast imaging

Olivier Flasseur, Théo Bodrito, Julien Mairal, Jean Ponce, M. Langlois, Anne-Marie Lagrange

<figure><img src="image_1.png" /></figure>

To cite this version:

Olivier Flasseur, Théo Bodrito, Julien Mairal, Jean Ponce, M. Langlois, et al. From statistical learning to deep learning for improved modeling of the nuisance component in high-contrast imaging. Exosystems IV 2024, Jan 2024, Lyon, France. hal-04792219

HAL Id: hal-04792219

https://hal.science/hal-04792219v1

Submitted on 20 Nov 2024

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_2.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# From statistical learning to deep learning for improved modeling of the nuisance component in high-contrast imaging

Olivier Flasseur¹, Théo Bodrito²

J. Mairal², J. Ponce², M. Langlois¹, A.-M. Lagrange³

<figure><img src="image_3.png" /></figure>

<figure><img src="image_4.png" /></figure>

<figure><img src="image_5.png" /></figure>

<figure><img src="image_6.png" /></figure>

<figure><img src="image_7.png" /><figcaption>Théo B. (PhD student)</figcaption></figure>

<figure><img src="image_8.png" /></figure>

<figure><img src="image_9.png" /></figure>

<figure><img src="image_10.png" /></figure>

<figure><img src="image_11.png" /></figure>
# Goal: modeling the nuisance component (speckles + noise)

## angular differential imaging (ADI) = temporal diversity

<figure><img src="image_12.png" /></figure>

## Peculiarities

* Faint signal from the exoplanets

* Non-stationary and spatially correlated strong nuisance component

* Strong fluctuations (stellar leakages)

* Multi-spectral data available

⇒ **Signal unmixing is critical**
# Goal: modeling the nuisance component (speckles + noise)

angular & spectral diff. im. (ASDI) = temporal & spectral diversity

<figure><img src="image_13.png" /></figure>

## Peculiarities

* Faint signal from the exoplanets

* Non-stationary and spatially correlated strong nuisance component

* Strong fluctuations (stellar leakages)

* Multi-spectral data available

⇒ **Signal unmixing is critical**
# Positioning within the post-processing landscape

**Subtraction - Decomposition:** cADI(Marois+ 2006); LOCI(Lafrénière+ 2007); ALOCI(Currie+ 2012); KLIP/PCA(Soummer+ 2012).(Amara+ 2012); TLOCI(Lafrénière+ 2014); LLSG(Gonzalez+ 16); NMF(Ren+ 18)

**Statistics:** MOODS(Smith+2009); ANDROMEDA(Cantalloube+ 2016); FMMF(Ruffio+ 2017); PACO(Flasseur+ 2020), etc.

**(Deep) Learning:** S4(Fergus+ 2014); (NA)SODINN(Gonzalez+ 2017).(Cantero+ 2023); CNN(Yip+ 2020)

**Temporal:** STIM(Pairet+ 2019); RSM(Dahlqvist+ 2020); TRAP(Samland+ 2021); HSR(Gebhard+ 2022)

**Physics:** PeX(Devaney+ 2017), MEDUSAE(Cantalloube+ 2018), etc.

<figure><img src="image_14.png" /></figure>
Pre-processing: removing (most of) the correlations

<figure><img src="image_15.png" /></figure>

centered & whitened observations

→ Contrast and stationarity are improved

→ Residual correlations will be captured by deep learning

<figure><img src="image_16.png" /></figure>

$\tilde{r}$
# Deep learning: capturing residual correlations

## properties of synthetic training sources

<figure><img src="image_17.png" /></figure>

## common processing

<figure><img src="image_18.png" /></figure>
# Detection pipeline

## Task: supervised pixel-wise classification

<figure><img src="image_19.png" /></figure>

## Loss function: custom similarity metric for very unbalanced classes

$$ \mathcal{L}(\mathbf{y}^{[s]}, \hat{\mathbf{y}}^{[s]}) = 1 - \frac{\underbrace{\sum_{m=1}^{M} \mathbf{y}_m^{[s]} \hat{\mathbf{y}}_m^{[s]} + \epsilon}_{\text{source error}}}{\sum_{m=1}^{M} \mathbf{y}_m^{[s]} + \hat{\mathbf{y}}_m^{[s]} + \epsilon} - \frac{\underbrace{\sum_{m=1}^{M} (1 - \mathbf{y}_m^{[s]}) (1 - \hat{\mathbf{y}}_m^{[s]} + \epsilon)}_{\text{background error}} $$

⇒ Dedicated data-augmentation + whitening + dedicated loss: deep model specialized for each datacube without overfitting
# Results: an example of inferred detection maps

## VLT/SPHERE-IRDIS data

star: HD 95086

<figure><img src="image_20.png" /></figure>

<figure><img src="image_21.png" /></figure>

<figure><img src="image_22.png" /></figure>

O true detection

<figure><img src="image_23.png" /></figure>

Δmissed detection

<figure><img src="image_24.png" /></figure>

☐false detection
# Results: improved detection sensitivity

## VLT/SPHERE-IRDIS data

star: HIP 88399

<figure><img src="image_25.png" /></figure>

⇒ additional gain up to 0.5 mag + ~optimal far from the star
# Results: joint multi-spectral processing

## Method

**Statistical learning:**

joint modeling of spatial and
spectral covariances.

**Deep learning:** training with
physical SED models (ExoREM).

<figure><img src="image_26.png" /><figcaption>IFS data</figcaption></figure>

SPHERE-IRDIS datasets

<figure><img src="image_27.png" /></figure>

SPHERE-IFS datasets

<table><thead><tr><th>PCA</th><th>stat. (PACO)</th><th>stat. + DL</th></tr></thead><tbody><tr><td>0.65</td><td>0.80</td><td><strong>0.96</strong></td></tr><tr><td>0.36</td><td>0.68</td><td><strong>0.76</strong></td></tr><tr><td>0.52</td><td>0.79</td><td><strong>0.85</strong></td></tr></tbody></table>

→ Better precision/recall trade-off than a posterior combination
Learning the nuisance from multiple observations

<figure><img src="image_28.png" /></figure>

⇒ Leveraging database diversity to build highly non-linear models

Data: part of SHINE survey

<figure><img src="image_29.png" /></figure>
# Comparison between model architectures

## observation-dependent model

<figure><img src="image_30.png" /></figure>

Common part: whitening
via estimation of spatial covariances

### Outputs

<figure><img src="image_31.png" /><figcaption>model (2): residual flux distribution cube</figcaption></figure>

<figure><img src="image_32.png" /></figure>

reconstruction loss (flux)
instead of detection loss

improved robustness

## observation-independent model

<figure><img src="image_33.png" /></figure>

deep models for
pre-processing
(speckles aligned)
AND
source detection
(object aligned)

correlations captured
at a larger spatial scale
# Results: inferred detection maps

<caption>## VLT/SPHERE-IRDIS data</caption>

parallactic rotation ≈ 23°

<figure><img src="image_34.png" /></figure>

Bodrito, Flasseur, Mairal, Ponce, Langlois, Lagrange, MNRS 2024 (https://arxiv.org/abs/2409.17178) 10/12
# Results: inferred detection maps

<caption>## VLT/SPHERE-IRDIS data</caption>

parallactic rotation ≈ 10°

<figure><img src="image_35.png" /></figure>

Bodrito, Flasseur, Mairal, Ponce, Langlois, Lagrange, MNRS 2024 (https://arxiv.org/abs/2409.17178)10/12
# Results: inferred detection maps

<caption>## VLT/SPHERE-IRDIS data</caption>

parallactic rotation ≈ 6°

<figure><img src="image_36.png" /></figure>
# Results: inferred detection maps

## VLT/SPHERE-IRDIS data

parallactic rotation ≈ 2°!

<figure><img src="image_37.png" /></figure>

<figure><img src="image_38.png" /></figure>

<figure><img src="image_39.png" /></figure>
# Results: a better precision / recall trade-off

## ROCs: mean results on VLT/SPHERE-IRDIS data

<figure><img src="image_40.png" /></figure>

⇒ Combined model (stat. + DL) leads to better fidelity
# Results: a better precision / recall trade-off

## ROCs: mean results on VLT/SPHERE-IRDIS data

<figure><img src="image_41.png" /></figure>

⇒ **Capturing speckle diversity and large scale correlation is helpful**

T. Bodrito, O. Flasseur+, in preparation
# Model ablation: importance of statistical model

## ROCs: mean results on VLT/SPHERE-IRDIS data

<figure><img src="image_42.png" /></figure>

⇒ Modeling statistically the covariances is critical!
## Take-home messages:

* Combining stat. & deep models improves precision-recall,

* Modeling the covariances is critical,

* Leveraging from observations archive bypasses the limit of ADI,

* A single model: once trained, inference is super fast.

⇒ **Could be useful for ELT's observations**

(highly structured nuisance, massive data, limited ADI)

## What's next?

* Nuisance's model is flexible → disk reconstruction.

* Integrating more explicitly direct image formation model $M$,

* joint estimation nuisance $s$ and objects $x$ with $\mathbf{obs} = s + Mx$

* solving an inverse problem with neural network as proximal operator

<figure><img src="image_43.png" /></figure>