# Physics-inspired generative adversarial modelling for fluctuation-based super-resolution microscopy

Hamza Mentagui, Luca Calatroni, Sébastien Schaub, Laure Blanc-Féraud

► To cite this version:

Hamza Mentagui, Luca Calatroni, Sébastien Schaub, Laure Blanc-Féraud. Physics-inspired generative adversarial modelling for fluctuation-based super-resolution microscopy. ISBI 2024 - 2024 IEEE International Symposium on Biomedical Imaging, May 2024, Athens, Greece. pp.1-4, 10.1109/ISBI56570.2024.10635791. hal-04306601

HAL Id: hal-04306601

https://hal.science/hal-04306601v1

Submitted on 25 Nov 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons CC BY 4.0 - Attribution - International License
PHYSICS-INSPIRED GENERATIVE ADVERSARIAL MODELLING FOR
FLUCTUATION-BASED SUPER-RESOLUTION MICROSCOPY

Luca Calatroni¹, Hamza Mentagui¹, Sébastien Schaub², and Laure Blanc-Féraud¹

¹ Université Côte d'Azur, CNRS, INRIA, I3S, France

² Sorbonne Université, CNRS, LBDV, France

ABSTRACT

Image super-resolution techniques exploiting the stochastic
fluctuations of image intensities have become a powerful tool
in fluorescence microscopy. Compared to other approaches,
these techniques can be applied under standard acquisition
settings and do not require special microscopes nor fluo-
rophores. Most of these approaches can be mathematically
modelled making use of second-order statistics possibly com-
bined with a priori regularisation on the desired solution. In
this work, we consider a different paradigm and formulate a
physical-inspired data-driven approach based on generative
learning. By simulating fluorescence and noise fluctuations
by means of a suitable double Poisson-type process, the
unknown distribution of the fluctuating sequence of low-
resolution and noisy images is approximated via a GAN-type
approach where both physical and network parameters are
optimised. In this work, we provide theoretical insights on
the choice of the corresponding cost functionals and gradient
computations, and assess practical performance on simulated
Argolight patterns.

***Index Terms***— Fluctuation-based super-resolution, fluo-
rescence microscopy, generative learning, inverse problems.

1. INTRODUCTION

Fluorescence microscopy (widefield, confocal) is limited
in resolution by light diffraction phenomena. Among the
many super-resolution methods developed in recent years
[1], super-resolution using molecular fluctuations has the
advantage of not requiring any particular fluorophores or
microscopes, and it is not harmful to the biological sample
under observation. The idea is to record a series of fluores-
cence images at a fairly fast rate (e.g. 100 images per second)
and to achieve super-resolution by analysing the fluctuations
of the fluorescent molecules which are independent in space
and time. Originally proposed in [2] for blinking-type fluc-
tuations in the popularised SOFI (super-resolution optical
fluctuation imaging), various approaches have then addressed
the analysis under more general fluctuation behaviour. Popu-
lar approaches are SRRF based on the estimation of radiality
map [3] or covariance-based methods [4, 5, 6].

Recently we have considered a physical-inspired data-
driven framework based on generative learning [7] for a de-
convolution problem where only noise fluctuations are simu-
lated. To achieve super-resolution, we consider in this work
a more refined model where both fluorescence and noise fluc-
tuations are modelled by means of a suitable double Poisson-
type stochastic process. The unknown distribution of the fluc-
tuating sequence of low-resolution and noisy images is ap-
proximated via a GAN-type approach where both physical
and network parameters are optimised. The main contribu-
tion of this work is to detail how a suitable cost functional
allowing to manipulate in a handy way the adversarial optimi-
sation process can be defined. We thus provide some insights
on the theoretical modelling and specify the approximation
techniques used to compute gradient updates. The proposed
algorithm is then tested on a small image of a simulated Ar-
golight slide showing the feasibility of the method to achieve
resolution up to 120 nm and first good results.

## 2. FORMULATION OF THE PROBLEM

We model a sequence of noisy and low-resolution images
with fluctuating intensities acquired over a time interval [0, T]
by vectors {$\mathbf{y}_t$}$_{t=1}^T$ with $\mathbf{y}_t \in \mathbb{R}^{m_1 m_2}$, $m_1, m_2 \in \mathbb{N}$ for all $t =
1, \dots, T$. We interpret such sequence as a set of independent
realisations of a stochastic process $\mathbf{Y}^{\text{real}}$ with unknown distri-
bution $\mathcal{D}$ which we aim to approximate by a physical-inspired
generator (simulator) drawing samples from a known distri-
bution $\mathcal{D}^{\text{sim}}$ parameterised by the super-resolved image we
want to retrieve $\mathbf{x} \in \mathbb{R}_{\ge 0}^{n_1 n_2}$ with $n_i = L m_i$ for $i \in \{1, 2\}$ and
$L > 0$ denoting the super-resolution factor and a space-variant
static background term $\mathbf{b} \in \mathbb{R}_{\ge 0}^{m_1 m_2}$. Denoting by $\mathbf{\Psi} = \mathbf{DH} \in$
$\mathbb{R}^{m_1 m_2 \times n_1 n_2}$ the matrix modelling the composition of the blur-
ring operator $\mathbf{H} \in \mathbb{R}^{n_1 n_2 \times n_1 n_2}$ related to the microscope PSF
with a down-sampling operator $\mathbf{D} \in \mathbb{R}^{m_1 m_2 \times n_1 n_2}$, we thus sim-
ulate fluctuations using the stochastic process:

$$
\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b}) \sim \mathcal{D}^{\text{sim}}(\mathbf{x}, \mathbf{b}) := \alpha \mathcal{P}(\mathbf{\Psi}\mathcal{P}(\mathbf{x}) + \mathbf{b}) + \mathbf{N}. \quad (1)
$$

The parameter $\alpha > 0$ is a gain parameter quantifying the
number of photons/emitter/frame which can be estimated
ahead. For $\mathbf{w} > 0$ we denote by $\mathcal{P}(\mathbf{w})$ a multidimensional
i.i.d. stochastic Poisson process of mean and variance $w_i$ w.r.t. the component $i$. In (1), such modelling is used both to describe fluorescence fluctuations (on the $n_1 \times n_2$ fine grid) and the photon-counting measurement process (on the $m_1 \times m_2$ coarse sensor grid). Note that in (1) vanishing parameters are excluded by adding a positive parameter $\epsilon > 0$. The additive noise random process $\mathbf{N} \sim \mathcal{N}(0, \sigma^2\mathbf{I})$ models signal-independent electronic noise. In a distributional sense, the inverse problem can thus be formulated as

$$ \text{find } (\mathbf{x}, \mathbf{b}) \in \mathbb{R}_{\ge 0}^{n_1 n_2} \times \mathbb{R}_{\ge 0}^{m_1 m_2} \quad \text{s.t.} \quad \mathcal{D}^{\text{sim}}(\mathbf{x}, \mathbf{b}) \approx \mathcal{D}, \quad (2) $$

where observed and generated samples ($\{\mathbf{y}_t\}$ and $\{\mathbf{y}_t^{\text{sim}}\}$, respectively) should thus be compared in terms of a suitable distance between probability distributions.

Standard approaches to (2) overcome the ill-posedness of the problem by minimising suitable cost functionals defined as the sum of data-fidelity and regularisation terms. In the context of fluctuation-based imaging, this has been done using fitting terms in the co-variance domain possibly combined with sparsity-promoting regularisations, see, e.g., [8, 9, 4, 6, 10] combined with optimisation-based learning approaches [5, 11]. While effective, hand-crafted approaches suffer from high-parameter sensitivity and possible reconstruction artefacts which are particularly visible when applied on real microscopy data characterised by heterogeneous (i.e., being a mixture of sparse and non-sparse) structures.

### 3. FLUOGAN MODELLING

Generative Adversarial Networks (GANs) were introduced firstly in [12]. The objective of a GAN is to approximate the (unknown) distribution of observed data, in our case $\mathcal{D}$, relying on training two competing networks, a generator network $G(\cdot)$ and a discriminator network $D_\varphi(\cdot) \in [0, 1]$ so that, at convergence, it is possible to generate samples with distribution $\mathcal{D}^{\text{sim}} \approx \mathcal{D}$. Generator and discriminator are thus trained together but with opposite goals. The class of plausible solutions output by $G$ can be restricted whenever prior information on the underlying physical model is available, thus, in principle, obtaining more interpretable solutions and less data to generate them. In the context of inverse problems, this idea has been firstly used for cryo-EM in [13] and exploited in [7] deconvolution in fluorescence microscopy. There, only Poisson/Gauss noise fluctuations were considered, with no modelling of fluorescence fluctuations of the signal $\mathbf{x}$ (i.e., the approximation $\mathcal{P}(\mathbf{x}) \approx \mathbf{x}$ was made). Here, under the more refined generative law (1), we consider the problems:

$$ \min_{\mathbf{x} \ge 0, \mathbf{b} \ge 0} \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [L_G(\mathbf{x}, \mathbf{b}, \varphi)], \quad \min_{\varphi} \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [L_D(\mathbf{x}, \mathbf{b}, \varphi)], \quad (3) $$

where for parameters $\gamma, \lambda \in \mathbb{R}_{\ge 0}$, the functionals read:

$$ \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [L_G(\mathbf{x}, \mathbf{b}, \varphi)] := \mathbb{E}_{\mathbf{Y}^{\text{sim}}} \left[ \frac{\gamma}{2} \|\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b}) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [\mathbf{Y}^{\text{real}}]\|^2 \right. \\ \left. - D_\varphi(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) \right] \quad (4) $$

$$ \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [L_D(\mathbf{x}, \mathbf{b}, \varphi)] := \mathbb{E}_{\mathbf{Y}^{\text{sim}}} \left[ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) - D_{\varphi}(\mathbf{Y}^{\text{real}}) \right. \\ \left. + \lambda \left( \|\nabla D_{\varphi}(\mathbf{Y}^{\text{mix}}(\mathbf{x}, \mathbf{b}))\| - 1 \right)^2 \right]. \quad (5) $$

In the spirit of [14, 15], a gradient penalty is introduced in (5) to enforce 1-Lipschitz regularity of $D_\varphi$ on its domain, which, as shown in [15] corresponds to minimise the Wasserstein distance between $\mathcal{D}^{\text{sim}}$ and $\mathcal{D}$. To do so the penalty is applied on $\mathbf{Y}^{\text{mix}}(\mathbf{x}, \mathbf{b}) := \eta \mathbf{Y}^{\text{real}} + (1 - \eta) \mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})$ with $\eta \sim \mathcal{U}([0, 1])$, with $\mathcal{U}([0, 1])$ being for the uniform distribution on $[0, 1]$.

Note that in (4) an $\ell^2$ term enforcing proximity between the generated data and the mean of the observed ones is introduced. This is unusual in standard GANs, but in [7] it was showed to stabilise convergence. In the following, we get better insights on this choice and propose a regularisation functional encoding physical constraints. To do so, we denote for simplicity by $\mathbf{Z} \sim \mathcal{P}(\mathbf{X})$ the random (since $\mathbf{X}$ is) process simulating fluorescence fluctuations and observe:

$$ \begin{aligned} & \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [\|\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b}) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [\mathbf{Y}^{\text{real}}]\|^2] \\ &= \mathbb{E}_{\mathbf{Z}} [\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [\|\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b}) - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b})]\|^2]] \\ & \quad + \mathbb{E}_{\mathbf{Z}} [\|\mathbb{E}_{\mathbf{Y}^{\text{sim}}} [\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b})] - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [\mathbf{Y}^{\text{real}}]\|^2], \end{aligned} \quad (6) $$

where $\hat{\mathbf{Y}}^{\text{sim}}$ is the stochastic process defined, for $\mathbf{z} \in \mathbb{R}^{n_1 n_2}$ and $\mathbf{b} \in \mathbb{R}^{m_1 m_2}$, by:

$$ \hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b}) \sim \alpha \mathcal{P}(\Psi\mathbf{z} + \mathbf{b}) + \mathbf{N}. \quad (7) $$

The right hand side in (6) decomposes the expectation in the sum of a variance and a bias term, respectively. As far as the first term is concerned, we notice that:

$$ \begin{aligned} & \mathbb{E}_{\mathbf{Z}} [\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [\|\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b}) - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b})]\|^2]] \\ &= \mathbb{E}_{\mathbf{Z}} \left[ \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} \left[ \sum_{j=1}^{m_1 m_2} \left( \hat{\mathbf{Y}}_{j}^{\text{sim}}(\mathbf{z}, \mathbf{b}) - \mathbb{E}_{\hat{\mathbf{Y}}_{j}^{\text{sim}}} [\hat{\mathbf{Y}}_{j}^{\text{sim}}(\mathbf{z}, \mathbf{b})] \right)^2 \right] \right] \\ &= \mathbb{E}_{\mathbf{Z}} \left[ \sum_{j=1}^{m_1 m_2} \text{Var} \left( \hat{\mathbf{Y}}_{j}^{\text{sim}}(\mathbf{z}, \mathbf{b}) \right) \right] = \mathbb{E}_{\mathbf{Z}} \left[ \sum_{j=1}^{m_1 m_2} \alpha^2 (\Psi\mathbf{z} + \mathbf{b})_j + \sigma^2 \right] \\ &= \mathbb{E}_{\mathbf{Z}} \left[ \alpha^2 \sum_{i=1}^{n_1 n_2} \|\Psi_i\|_1 z_i \right] + \alpha^2 \|\mathbf{b}\|_1 + m_1 m_2 \sigma^2 \\ &= \alpha^2 \left( \sum_{i=1}^{n_1 n_2} \|\Psi_i\|_1 x_i + \|\mathbf{b}\|_1 \right) + m_1 m_2 \sigma^2, \end{aligned} \quad (8) $$

where $\Psi_i$, $i = 1, \dots, n_1 n_2$ denotes the $i$-th column of the matrix $\Psi$ and computations follow by simple algebraic manipulations, the fact $\Psi_i, \mathbf{b}$ are non-negative and on the property $\mathbb{E}_{\mathbf{Z}}[\mathbf{Z}] = \mathbf{x}$ which holds by definition of Poisson process.
As far as the bias term is concerned we have:

$$
\begin{align}
& \mathbb{E}_Z [ \| \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [ \hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b}) ] - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [ \mathbf{Y}^{\text{real}} ] \|^2 ] \nonumber \\
&= \mathbb{E}_Z [ \| \alpha ( \mathbf{\Psi}\mathbf{z} + \mathbf{b} ) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [ \mathbf{Y}^{\text{real}} ] \|^2 ] \nonumber \\
&\quad + \| \alpha ( \mathbf{\Psi}\mathbf{x} + \mathbf{b} ) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [ \mathbf{Y}^{\text{real}} ] \|^2 . \tag{9}
\end{align}
$$

Denoting by $w_{i,j} = (\Psi^T \Psi)_{i,j}$, simple calculations show:

$$
\mathbb{E}_Z [ \| \mathbf{\Psi}(\mathbf{z} - \mathbf{x}) \|^2 ] = \sum_{i=1}^{n_1 n_2} \mathbf{v} \mathbf{a}(z_i) w_{ii} + 2 \sum_{i<j} \mathbf{C} \mathbf{o}(z_i, z_j) w_{ij} = \sum_{i=1}^{n_1 n_2} \| \mathbf{\Psi}_i \|_2^2 x_i,
$$

so that combining (8) and (9), we get from (6):

$$
\begin{align}
& \mathbb{E}_{\mathbf{Y}^{\text{sim}}} [ \| \mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b}) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [ \mathbf{Y}^{\text{real}} ] \|^2 ] \nonumber \\
&= \| \alpha ( \mathbf{\Psi}\mathbf{x} + \mathbf{b} ) - \mathbb{E}_{\mathbf{Y}^{\text{real}}} [ \mathbf{Y}^{\text{real}} ] \|^2 \nonumber \\
&\quad + \alpha^2 \left( \sum_{i=1}^{n_1 n_2} ( \| \mathbf{\Psi}_i \|_1 + \| \mathbf{\Psi}_i \|_2^2 ) |x_i| + \| \mathbf{b} \|_1 \right) + m_1 m_2 \sigma^2. \tag{10}
\end{align}
$$

Expansion (10) shows that an $\ell^1$ type regularisation on both the desired reconstructed image and the background is thus naturally enforced. However, on $\mathbf{x}$, we notice that the regularisation weight at each pixel depends on the value of $\|\mathbf{\Psi}_i\|_2^2 \le \|\mathbf{\Psi}_i\|_1 \le 1$ as they consist of only a subset of the elements defining the PSF. This is undesirable as in the case of large noise a higher regularisation weighting could be preferable. Moreover, the sparsity constraint on $\mathbf{b}$ does not fit most of the real-world applications where background encodes smooth intensity variations describing out-of-focus and/or auto-fluorescence.

To overcome this, we thus make some modifications in (10) so as to allow for larger sparse regularisation on $\mathbf{x}$ and to encode smoothness on $\mathbf{b}$, while removing the explicit dependence on $\alpha$. Recalling (4) and for $\gamma, \lambda_1, \lambda_2 \ge 0$ we thus consider the functional:

$$
\begin{align}
\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ L_G (\mathbf{x}, \mathbf{b}, \varphi) ] &= \frac{\gamma}{2} \| \mathbf{\Psi}\mathbf{x} + \mathbf{b} - \mathbb{E}[\mathbf{Y}^{\text{real}}] \|^2 \nonumber \\
&\quad - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ] + \lambda_1 \|\mathbf{x}\|_1 + \frac{\lambda_2}{2} \|\nabla\mathbf{b}\|^2, \tag{11}
\end{align}
$$

which we use in (3) together with $L_D$ in (5) for training. Note that the last term in (10) can be neglected as we assume $\sigma^2$ to be known/estimated in advance.

4. OPTIMISATION INSIGHTS

For the minimisation problems in (3) under the choice (5) and (11) suitable algorithms can be used. In particular, due to the convenient structure (11), the computation of $\nabla_{\mathbf{x}}\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}}[L_G(\mathbf{x}, \mathbf{b}, \varphi)]$ (and similarly of $\nabla_{\mathbf{b}}$) can be done by using standard gradient/proximal operators for all terms but $\nabla_{\mathbf{x}}\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}}[D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b}))]$. To deal with it, we write:

$$
\nabla_{\mathbf{x}} \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ] = \nabla_{\mathbf{x}} \mathbb{E}_{\mathbb{Z}} [ h(\mathbf{z}, \mathbf{b}) ],
$$

with $h(\mathbf{z}, \mathbf{b}) := \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b})) ]$, and then follow similar arguments as in [7] to obtain:

$$
\nabla_{\mathbf{x}} \mathbb{E}_{\mathbb{Z}} [ h(\mathbf{z}, \mathbf{b}) ] = \mathbb{E}_{\mathbb{Z}} [ \{ h(\mathbf{z} + \mathbf{e}_i, \mathbf{b}) - h(\mathbf{z}, \mathbf{b}) \}_{i=1, \dots, n_1 n_2} ], \quad (12)
$$

where $\mathbf{e}_i$ is the $i$-th vector of the canonical basis in $\mathbb{R}^{n_1 n_2}$.
Since by definition (7) there trivially holds:

$$
\mathbf{Y}^{\text{sim}}(\mathbf{z}+\mathbf{e}_i, \mathbf{b}) \sim \alpha \mathcal{P}(\mathbf{\Psi}(\mathbf{z}+\mathbf{e}_i)+\mathbf{b})+\mathbf{N} = \alpha \mathcal{P}(\mathbf{\Psi}\mathbf{z}+\mathbf{C}_i+\mathbf{b})+\mathbf{N},
$$

with $\mathbf{C}_i$ being the $i$-th column of $\mathbf{\Psi}$, we have $\mathbf{Y}^{\text{sim}}(\mathbf{z}+\mathbf{e}_i, \mathbf{b}) = \mathbf{Y}^{\text{sim}}(\mathbf{z}, \mathbf{b}+\mathbf{C}_i)$. Carrying on calculations in (12) we thus get:

$$
\begin{align*}
\nabla_{\mathbf{x}} \mathbb{E}_{\mathbb{Z}} [ h(\mathbf{z}, \mathbf{b}) ] &= \mathbb{E}_{\mathbb{Z}} [ \{ \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b} + \mathbf{C}_i)) \\
&\quad - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\hat{\mathbf{Y}}^{\text{sim}}(\mathbf{z}, \mathbf{b})) ] \}_{i=1, \dots, n_1 n_2} ] \\
&= \{ \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b} + \mathbf{C}_i)) ] - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ] \}_{i=1, \dots, n_1 n_2}
\end{align*}
$$

Proceeding similarly for $\nabla_{\mathbf{b}}\mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ]$, we get:

$$
\begin{align*}
\nabla_{\mathbf{b}} \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ] \\
&= \{ \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b} + \mathbf{e}_i)) ] - \mathbb{E}_{\hat{\mathbf{Y}}^{\text{sim}}} [ D_{\varphi}(\mathbf{Y}^{\text{sim}}(\mathbf{x}, \mathbf{b})) ] \}_{i=1, \dots, m_1 m_2}.
\end{align*}
$$

Thanks to these computations we can now make precise the type of algorithms used for minimising (3) under the choice (5) and (11). We alternate minimisation over $(\mathbf{x}, \mathbf{b})$ for (11) and over the parameter network $\varphi$ of the discriminator $D_{\varphi}$ for (5). As far as the optimisation of (11), we consider block-coordinate proximal gradient descent, where the gradient terms are computed explicitly and using the expressions above, while non-negativity constraints and, in the case of $\mathbf{x}$, the sparsity-promoting regularisation is enforced using a ReLu (i.e. a soft-thresholding operator restricted to the non-negative orthant). Optimisation of (5) is performed using ADAM [16].

5. NUMERICAL RESULTS

We validate FluoGAN on a sequence of $T = 500$ of data $\{y_t\}$ generated using the SOFI simulator [17], a tool developed to simulated realistic fluctuations based on a given spatial pattern, see Fig. 1a) for the image $\bar{y}$. With the future intent of testing our model on real Argolight slides, we simulated a spatial ground-truth spatial pattern $\mathbf{x}^* \in \mathbb{R}^{n_1 \times n_2}$ composed by a sequence of parallel lines which are 100 nm wide. The separation distance $d$ (centre-to-centre distance) between the two middle lines of each set is gradually decreasing from 390 nm up to 120 nm with a rate of 30nm, see Fig. 1b) for a detail at $d = 120$nm. To simulate realistic acquisitions we provided SOFI tool with a smoothly varying background image $\mathbf{b}^* \in \mathbb{R}^{m_1 \times m_2}$, see 1c). Note that in Fig. 1a) inner lines can hardly be distinguished at all distances $d$. We apply FluoGAN to reconstruct the small ROI highlighted in red. The physical inspired simulator (1) requires also the use of physical parameters describing the microscope PSF, which we estimated
to be of FWHM=318 nm. We fixed an undersampling factor $L = 4$. The standard deviation $\sigma$ of the Gaussian noise component was provided beforehand. Regarding the structure of $D_{\varphi}$, we consider a convolutional neural network (CNN) with 3 convolutional layers performing convolutions of size 3 with max pooling and with 32 initial channels. A careful selection of the step-size parameters $\tau_x$, $\tau_b$ and $\tau_{\varphi}$ must be performed to guarantee convergence. Regularisation parameters $\gamma$, $\lambda_1$, $\lambda_2$ was chosen manually. Training is performed for 5000 epochs with mini-batches of 200 images.

The numerical reconstruction obtained shows that the two inner lines can be indeed be reconstructed at a distance $d = 120$ nm, up to some reconstruction biases induced by the use of the $\ell_1$ norm as a regulariser. Further tests and validation on more challenging real data have to be performed to assess the robustness of the approach.

<figure><img src="image_2.png" /><figcaption>(a) Temporal mean $\bar{y}$ of fluctuating stack simulated by SOFI tool.</figcaption></figure>

<figure><img src="image_3.png" /><figcaption>(b) GT, $d = 120$nm.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>(c) Simulated $\mathbf{b}^*$.</figcaption></figure>

<figure><img src="image_5.png" /><figcaption>(c) FluoGAN result.</figcaption></figure>

<caption>**Fig. 1:** Reconstruction of data generated using the SOFI tool [17] using Arolight slide-type patterns.</caption>

# 6. CONCLUSION

In this paper we proposed a precise model to generate fluorescence fluctuations data by means of a suitable double Poisson-type stochastic process and use it as a generator in a GAN approach. The super-resolved image **x** and the background image **b** are estimated to produce samples with a distribution $\mathcal{D}^{\text{sim}}$ close to the experimental distribution $\mathcal{D}$ in the Wasserstein distance sense. The computation of the objective function and its gradient is provided and a first numerical result shows the pertinence of the approach. Further simulations as well as real data on a real Argolight slide are underway and will be compared to state of the art methods.

# 7. COMPLIANCE WITH ETHICAL STANDARDS

Not concerned, simulated data only.

# 8. ACKNOWLEDGEMENTS

The authors acknowledge the support by the ANR MICROBLIND grant ANR-21-CE48-0008 and the ANR TASKABILE JCJC grant ANR-22-CE48-0010. The work of LBF has been supported by the French government,

through the 3IA Côte d'Azur Investments in the Future project managed by the National Research Agency, grant ANR-19-P3IA-0002.

# 9. REFERENCES

[1] T. Huser L. Schermelleh, A. Ferrand et al., "Super-resolution microscopy demystified," *Nature Cell biology*, vol. 21, Jan 2019.

[2] T. Dertinger, R. Colyer, G. Iyer, S. Weiss, and J. Enderlein, "Fast, background-free, 3d super-resolution optical fluctuation imaging (sofi)," *Proc. of the Nat. Acad. of Sciences*, vol. 106, no. 52, 2009.

[3] N. Gustafsson, S. Culley, G. Ashdown, et al., "Fast live-cell conventional fluorophore nanoscopy with imagej through super-resolution radial fluctuations," *Nat Com*, vol. 7, no. 12471, March 2016.

[4] O. Solomon, Y. C. Eldar, M. Mutzafi, and M. Segev, "Sparcom: Sparsity based super-resolution correlation microscopy," *SIAM Journal on Imaging Sciences*, vol. 12, no. 1, pp. 392-419, 2019.

[5] G. Dardikman-Yoffe and Y. C. Eldar, "Learned sparcom: unfolded deep super-resolution microscopy," *Opt. Express*, vol. 28, no. 19, pp. 27736-27763, Sep 2020.

[6] V. Stergiopoulou, J. H. de M. Goulart, S. Schaub, L. Calatroni, and L. Blanc-Féraud, "COLORME: Covariance-based $\ell_0$ super-resolution microscopy with intensity estimation," in *2021 IEEE ISBI*, 2021.

[7] M. Cachia, V. Stergiopoulou, L. Calatroni, S. Schaub, and L. Blanc-Féraud, "Fluorescence image deconvolution microscopy via generative adversarial learning (FluoGAN)," *Inverse Problems*, vol. 39, no. 5, pp. 054006, apr 2023.

[8] S. Labouesse, A. Negash, J. Idier, S. Bourguignon, T. Mangeat, P. Liu, A. Sentenac, and M. Allain, "Joint reconstruction strategy for structured illumination microscopy with unknown illuminations," *IEEE Transactions on Image Processing*, vol. 26, no. 5, pp. 2480-2493, 2017.

[9] J. Idier, S. Labouesse, M. Allain, P. Liu, S. Bourguignon, and A. Sentenac, "On the superresolution capacity of imagers using unknown speckle illuminations," *IEEE Transactions on Computational Imaging*, vol. 4, no. 1, pp. 87-98, 2018.

[10] V. Stergiopoulou, L. Calatroni, J. H. de M. Goulart, S. Schaub, and L. Blanc-Féraud, "Colorme: Super-resolution microscopy based on sparse blinking/fluctuating fluorophore localization and intensity estimation," *Biological Imaging*, vol. 2, 2022.

[11] V. Stergiopoulou, S. Mukherjee, L. Calatroni, and L. Blanc-Féraud, "Fluctuation-based deconvolution in fluorescence microscopy using plug-and-play denoisers," in *SSVM*, 2023, pp. 498-510, Springer.

[12] I. Goodfellow et al., "Generative adversarial nets," in *Advances in Neural Information Processing Systems*, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence, and K.Q. Weinberger, Eds. 2014, vol. 27, Curran Associates, Inc.

[13] H. Gupta, M. T. McCann, L. Donati, and M. Unser, "CryoGAN: A New Reconstruction Paradigm for Single-Particle Cryo-EM Via Deep Adversarial Learning," *IEEE TCI*, vol. 7, 2021.

[14] M. Arjovsky, S. Chintala, and L. Bottou, "Wasserstein generative adversarial networks," in *Proceedings of ICML*, 2017, pp. 214-223.

[15] I. Gulrajani, F. Ahmed, M. Arjovsky, V. Dumoulin, and A. C. Courville, "Improved training of Wasserstein GANs," *CoRR*, vol. abs/1704.00028, 2017.

[16] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," in *ICLR*, 2015.

[17] A. Girault, T. Lukes, A. Sharipov, et al., "SOFI simulation tool: A software package for simulating and testing super-resolution optical fluctuation imaging," *PLOS ONE*, vol. 11, no. 9, pp. 1-13, 2016.