\documentclass[runningheads]{llncs}
\usepackage{graphicx}
\usepackage{soul}
\usepackage[normalem]{ulem}
\usepackage{tikz}
\usepackage{comment}
\usepackage{amsmath,amssymb} %
\usepackage{color}
\usepackage[misc]{ifsym}
\usepackage{xspace}
\usepackage{tabularx}
\usepackage{multirow}
\newcommand{\miniheadline}[1]{\noindent\textbf{#1.}}

\newcommand\todo[1]{\textcolor{red}{TODO: #1}}

\makeatletter
\DeclareRobustCommand\onedot{\futurelet\@let@token\@onedot}
\def\@onedot{\ifx\@let@token.\else.\null\fi\xspace}

\def\eg{\emph{e.g}\onedot} \def\Eg{\emph{E.g}\onedot}
\def\ie{\emph{i.e}\onedot} \def\Ie{\emph{I.e}\onedot}
\def\cf{\emph{c.f}\onedot} \def\Cf{\emph{C.f}\onedot}
\def\etc{\emph{etc}\onedot} \def\vs{\emph{vs}\onedot}
\def\wrt{w.r.t\onedot} \def\dof{d.o.f\onedot}
\def\etal{\emph{et al}\onedot}
\makeatother
\newcommand{\E}[2]{\mathbb{E}_{#2} {\left[ #1 \right]} }
\newcommand{\KL}[2]{\mathbb{KL}(#1||#2)}
\newcommand{\oursm}{ours{\tiny{$^-$}}}
\newcommand{\oursp}{ours{\tiny{$^+$}}}
\newcommand{\VAE}{\mbox{\textsc{VAE}}\xspace}
\newcommand{\VAEs}{\mbox{\textsc{VAE}}s\xspace}
\newcommand{\CARE}{\mbox{\textsc{CARE}}\xspace}
\newcommand{\CSBDeep}{\mbox{\textsc{CSBDeep}}\xspace}
\newcommand{\NoiseNoise}{\mbox{\textsc{Noise2Noise}}\xspace}
\newcommand{\NoiseVoid}{\mbox{\textsc{Noise2Void}}\xspace}
\newcommand{\NoiseSelf}{\mbox{\textsc{Noise2Self}}\xspace}
\newcommand{\DenoiSeg}{\mbox{\textsc{DenoiSeg}}\xspace}
\newcommand{\DivNoising}{\mbox{\textsc{DivNoising}}\xspace}
\newcommand{\NtoN}{\mbox{\textsc{N2N}}\xspace}
\newcommand{\NtoV}{\mbox{\textsc{N2V}}\xspace}
\newcommand{\PNtoV}{\mbox{\textsc{PN2V}}\xspace}
\newcommand{\PNtoVgmm}{\mbox{\textsc{PN2V-GMM}}\xspace}
\newcommand{\PNtoVhist}{\mbox{\textsc{PN2V-H}}\xspace}
\newcommand{\UNet}{\mbox{\textsc{U-Net}}\xspace}
\newcommand{\imgp}{x}
\newcommand{\sigp}{s}
\newcommand{\sigpe}{\hat{s}}
\newcommand{\img}{\mathbf{x}}
\newcommand{\sig}{\mathbf{s}}
\newcommand{\sige}{\hat{\mathbf{s}}}
\newcommand{\seg}{\mathbf{c}}
\newcommand{\loss}[1]{\mathcal{L}_{\pars}{(#1)}}
\newcommand{\losskl}[1]{\mathcal{L}_\encopas^\textsc{KL}{(#1)}}
\newcommand{\lossr}[1]{\mathcal{L}_{\encopas,\decopas}^\textsc{R}{(#1)}}
\newcommand{\recf}{\img^\textsc{RF}}

\newcommand{\latente}{\hat{\mathbf{z}}}
\newcommand{\latentpe}{\hat{z}}
\newcommand{\sample}{\mathbf{s}}
\newcommand{\latent}{{\mathbf{z}}}
\newcommand{\psf}{{\mathbf{h}}}
\newcommand{\latentp}{z}
\newcommand{\encopas}{{\mathbf{\phi}}}
\newcommand{\enc}[1]{f_\encopas(#1)}
\newcommand{\pars}{{\mathbf{\theta} }}
\newcommand{\dec}[1]{g_\decopas(#1)}
\newcommand{\q}[1]{q_{\encopas}(#1)}
\newcommand{\p}[1]{p(#1)}
\newcommand{\pt}[1]{p_{\decopas}(#1)}
\newcommand{\pnm}[1]{p_\textsc{NM}(#1)}
\newcommand{\numpix}{N}
\newcommand{\numimgs}{M}
\newcommand{\numsamples}{K}
\newcommand{\numlatdim}{D}
\newcommand{\setRandPix}{M}
\newcommand{\MMSE}{\textsc{MMSE}\xspace}
\newcommand{\MAP}{\textsc{MAP}\xspace}
\newcommand{\GMM}{\textsc{GMM}\xspace}
\newcommand{\PSF}{\textsc{PSF}\xspace}
\newcommand{\PSFs}{\textsc{PSF}s\xspace}
\newcommand{\SURE}{\textsc{SURE}\xspace}
\usepackage{booktabs}
\usepackage{multirow}
\usepackage[normalem]{ulem}
\useunder{\uline}{\ul}{}
\newcommand\figSchema{
\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figs/network.pdf}
    \caption{
    \textbf{Improved Denoising for Diffraction-Limited Data.}
    \textbf{Top:} 
    Given a noisy input, self-supervised methods like \NoiseVoid (N2V)~\cite{krull2019noise2void} often produce high-frequency artifacts that do not occur in diffraction-limited data.
    Based on the assumption that the true signal must be the product of a convolution with a \emph{point spread function} (\PSF), our method is able to considerably improve denoising quality and remove these artifacts.
    \textbf{Bottom:}
    Our method is based on the \NoiseVoid masking scheme.
    Unpaired training images simultaneously serve as input and target.
    The loss is only calculated for a randomly selected set of pixels, which are masked in the input image.
    Our contribution is to convolve the output of the network with the \PSF in order to produce a denoising result that is guaranteed to be consistent with diffraction-limited imaging.
    The output of the network before the convolution operation can be interpreted as a deconvolution result, which is a byproduct of our method.
    Our system can be trained in an end-to-end fashion, calculating the loss between our denoising result and the selected pixel set of the input image.
    }
    \label{fig:schema}
\end{figure}
}

\newcommand\figTable{
\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figs/example_results_table_v1.pdf}
    \caption{
    \textbf{Denoising results.}
    We show cropped denoising results for various fluorescence microscopy datasets.
    Our method achieves considerable visual improvements for all datasets compared to \NoiseVoid.
    The \emph{N2V~(conv.)} baseline corresponds to the \NoiseVoid result convolved with the same \PSF we use for our proposed method.
    }
    \label{fig:table}
\end{figure}
}

\newcommand\figDeconv{
\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figs/deconvolution_results_examples.pdf}
    \caption{\textbf{Effect of the proposed Positivity Constraint.}
    We show cropped denoising and deconvolution results from various datasets with (\emph{\oursp}) and without positivity constraint (\emph{\oursm}), see Section~\ref{sec:posConstr} for details. 
    While the denoising results are almost indistinguishable, the deconvolution results show a drastic reduction of artifacts when the positivity constraint is used.
    }
    \vspace{-2mm}
    \label{fig:deconv}
\end{figure}
}
\newcommand\figPSF{
\begin{figure}[h!]
    \centering
    \includegraphics[width=1\linewidth]{figs/psf_text.pdf}
    \caption{
    \textbf{Effects of Point Spread Function Mismatch.}
    We use synthetic data to investigate how the choice of \PSF influences the resulting denoising quality.
    The data was generated by convolving rendered text with a Gaussian \PSF of standard deviation $\sigma=1$ (highlighted in red) and subsequently adding noise.
    Here, we show the results of our method when trained using Gaussian \PSFs of various sizes.
    We achieve the best results by using the true \PSF.
    Smaller \PSFs produce high-frequency artifacts.
    Larger \PSFs produce overly smooth images.
    }
    \vspace{-2mm}
    \label{fig:psf}
\end{figure}
}

\newcommand\tablePSNR{
\begin{table}[]
\centering
\begin{tabular}{|l|c|cccc|cc|c|}
\hline
\multicolumn{1}{|c|}{\multirow{3}{*}{\begin{tabular}[c]{@{}c@{}}dataset/\\ network\end{tabular}}} & \multirow{3}{*}{raw data} & \multicolumn{6}{c|}{self-supervised}                                                                                                                        & \multirow{2}{*}{superv.} \\ \cline{3-8}
\multicolumn{1}{|c|}{}                                                                            &                           & \multicolumn{4}{c|}{no noise model}                                                                                      & \multicolumn{2}{c|}{noise model} &                          \\ \cline{3-9} 
\multicolumn{1}{|c|}{}                                                                            &                           & N2V   & \multicolumn{1}{l}{\begin{tabular}[c]{@{}l@{}}N2V \\ conv.\end{tabular}} & ours$^-$,            & ours$^+$       & PN2V         & DivN.             & CARE                     \\ \hline
Convallaria                                                                                       & 28.98                     & 35.85 & 32.86                                                                    & \textbf{36.39}       & 36.26          & 36.47        & {\ul 36.94}       & 36.71                    \\
Mouse actin                                                                                       & 23.71                     & 33.35 & 33.48                                                                    & 33.94                & \textbf{34.04} & 33.86        & 33.98             & {\ul 34.20}              \\
Mouse nuclei                                                                                      & 28.10                     & 35.86 & 34.59                                                                    & \textbf{36.34}       & 36.27          & 36.35        & 36.31             & {\ul 36.58}              \\
Flywing (DenoiSeg)                                                                                       & 11.15                     & 23.62 & 23.51                                                                    & 24.10                & \textbf{24.30} & 24.85        & 25.10             & {\ul 25.60}              \\
Mouse (DenoiSeg)                                                                                        & 20.84                     & 33.61 & 32.27                                                                    & \textbf{33.91}       & 33.83          & 34.19        & 34.03             & {\ul 34.63}              \\
W2S avg1 ch0                                                                                      & 21.86                     & 34.30 & 34.38                                                                    & {\ul \textbf{34.90}} & 34.24          & -            & 34.13             & 34.30                    \\
W2S avg1 ch1                                                                                      & 19.35                     & 31.80 & 32.23                                                                    & {\ul \textbf{32.31}} & 32.24          & -            & 32.28             & 32.11                    \\
W2S avg1 ch2                                                                                      & 20.43                     & 34.65 & {\ul \textbf{35.19}}                                                     & 35.03                & 35.09          & 32.48        & 35.18             & 34.73                    \\
W2S avg16 ch0                                                                                     & 33.20                     & 38.80 & 38.73                                                                    & \textbf{39.17}       & 37.84          & 39.19        & 39.62             & {\ul 41.94}              \\
W2S avg16 ch1                                                                                     & 31.24                     & 37.81 & 37.49                                                                    & \textbf{38.33}       & 38.19          & 38.24        & 38.37             & {\ul 39.09}              \\
W2S avg16 ch2                                                                                     & 32.35                     & 40.19 & 40.32                                                                    & 40.60                & \textbf{40.74} & 40.49        & 40.52             & {\ul 40.88}              \\ \hline
\end{tabular}
\vspace{.3cm}
\caption{\textbf{Quantitative Denoising Results.}
We report the average peak signal to noise ratio for each dataset and method.
Here, \textit{\oursp} and \textit{\oursm} correspond to our method with ($\lambda=1$) and without  positivity constraint ($\lambda=0$), see Section~\ref{sec:posConstr} for details. 
The best results among self-supervised methods without noise model are highlighted in bold.
The best results overall are underlined.
Here \emph{DivN.} is short for \DivNoising~\cite{prakash2020divnoising}.
}
\label{tab:results}
\end{table}
}
\begin{document}
\pagestyle{headings}
\mainmatter

\title{Improving Blind Spot Denoising\\ for Microscopy} %
\author{Anna~S.~Goncharova\inst{1,2} \and
Alf~Honigmann\inst{1} \and
Florian~Jug\inst{1,2,3, \text{\Letter}} \and
Alexander~Krull\inst{1,2,4, \text{\Letter}}}
\authorrunning{A. Goncharova et al.}

\institute{Max-Planck Institute of Molecular Cell Biology and Genetics, Dresden, Germany \and
Center for Systems Biology Dresden (CSBD), Dresden, Germany \and
Fondazione Human Technopole, Milano, Italy \and 
Max Planck Institute for the Physics of Complex Systems, Dresden, Germany\\
\Letter \: \text{jug@mpi-cbg.de}, \text{krull@mpi-cbg.de}}

\maketitle

\begin{abstract}
Many microscopy applications are limited by the total amount of usable light and are consequently challenged by the resulting levels of noise in the acquired images.
This problem is often addressed via (supervised) deep learning based denoising. 
Recently, by making assumptions about the noise statistics, self-supervised methods have emerged.
Such methods are trained directly on the images that are to be denoised and do not require additional paired training data.
While achieving remarkable results, self-supervised methods can produce high-frequency artifacts and achieve inferior results compared to supervised approaches.
Here we present a novel way to improve the quality of self-supervised denoising.
Considering that light microscopy images are usually diffraction-limited, we propose to include this knowledge in the denoising process.
We assume the clean image to be the result of a convolution with a point spread function (PSF) and explicitly include this operation at the end of our neural network.
As a consequence, we are able to eliminate high-frequency artifacts and achieve self-supervised results that are very close to the ones achieved with traditional supervised methods.
\keywords{denoising, CNN, light microscopy, deconvolution}
\end{abstract}

\figSchema
\section{Introduction}
For most microscopy applications, finding the right exposure and light intensity to be used involves a trade-off between maximizing the signal to noise ratio and minimizing undesired effects such as phototoxicity.
As a consequence, researchers often have to cope with considerable amounts of noise.
To mitigate this issue, denoising plays an essential role in many data analysis pipelines, enabling otherwise impossible experiments~\cite{belthangady2019applications}.

Currently, deep learning based denoising, also known as content-aware image restoration (\CARE)~\cite{weigert2018content}, achieves the highest quality results.
\CARE methods learn a mapping from noisy to clean images.
Before being applied, they must be trained with pairs of corresponding noisy and clean training data.

In practice, this dependence on training pairs can be a bottleneck.
While noisy images can usually be produced in abundance, recording their clean counterparts is difficult or impossible.

Over the last years, various solutions to the problem have been proposed.
Lehtinen \etal showed that a network can be trained for denoising using only pairs of corresponding noisy images.
This method is known as \NoiseNoise~\cite{lehtinen2018noise2noise}.

The first self-supervised approaches \NoiseVoid~\cite{krull2019noise2void} and \NoiseSelf~\cite{batson2019noise2self} were introduced soon after this.
These methods can be trained on unpaired noisy image data.
In fact, they can be trained on the very same data that is to be denoised in the first place.
The underlying approach relies on the assumption that (given the true signal) the noise in an image is generated independently for each pixel, as is indeed the case for the dominant sources of noise in light microscopy (Poisson shot noise and Gaussian readout noise)~\cite{luisier2010image,zhang2019poisson}.
Both methods employ so-called \emph{blind spot} training,
in which random pixels are masked in the input image with the network trying to predict their value from the surrounding patch.

Unfortunately, the original self-supervised methods typically produce visible high-frequency artifacts (see Figure~\ref{fig:schema}) and can often not reach the quality achieved by supervised \CARE training.
It is worth noting that the high-frequency artifacts produced by these self-supervised methods never occur in the real fluorescence signal.
Since the image is diffraction-limited and oversampled, the true signal has to be smooth to some degree.

Multiple extensions of \NoiseVoid and \NoiseSelf have been proposed~\cite{Krull:2020_PN2V,laine2019high,Prakash2019ppn2v,khademi2020self}.
All of them improve results by explicitly modeling the noise distribution.

Here, we propose an alternate and novel route to high-quality self-supervised denoising.
Instead of making additional assumptions about the noise, we show that the result can be improved by including additional knowledge about the structure of our signal.
We believe that our approach might ultimately complement existing methods that are based on noise modeling, to further improve denoising quality.

We assume that the true signal is the product of a convolution of an unknown \emph{phantom image} and an approximately known point spread function (PSF) -- a common assumption in established deconvolution approaches~\cite{richardson1972bayesian}.
We use a \UNet~\cite{ronneberger2015u} to predict the phantom image and then explicitly perform the convolution to produce the final denoised result (see Figure~\ref{fig:schema}).
We follow~\cite{krull2019noise2void,batson2019noise2self} and use a blind spot masking scheme allowing us to train our network in an end-to-end fashion from unpaired noisy data.

We demonstrate that our method achieves denoising quality close to supervised methods on a variety of real and publicly available datasets.
Our approach is generally on-par with modern noise model based methods~\cite{Krull:2020_PN2V,prakash2020divnoising}, while relying on a much simpler pipeline.

As a byproduct, our method outputs the predicted phantom image, which can be interpreted as a deconvolution result.
While we focus on the denoising task in this paper, we find that we can produce visually convincing deconvolved images by including a positivity constraint for the deconvolved output.

\section{Related work}
\label{sec:relatedWork}
In the following, we will discuss related work on self-supervised blind spot denoising and other unsupervised denoising methods.
We will focus on deep learning-based methods and omit the more traditional approaches that directly operate on individual images without training.
Finally, we will briefly discuss concurrent work that tries to jointly solve denoising and inverse problems such as deconvolution.

\subsection{Self-Supervised Blind Spot Denoising}
By now, there is a variety of different blind spot based methods.
While the first self-supervised methods (\NoiseVoid and \NoiseSelf) use a masking scheme to implement blind spot training, Laine \etal~\cite{laine2019high} suggest an alternative approach.
Instead of masking, the authors present a specific network architecture that directly implements the blind spot receptive field.
Additionally, the authors proposed a way to improve denoising quality by including a simple pixel-wise Gaussian based noise model.
In parallel, Krull \etal~\cite{Krull:2020_PN2V} introduced a similar noise model based technique for improving denoising quality, this time using the pixel masking approach.
Instead of Gaussians, Krull~\etal use histogram-based noise models together with a sampling scheme.
Follow-up work additionally introduces parametric noise models and demonstrates how they can be bootstrapped (estimated) directly from the raw data~\cite{Prakash2019ppn2v}.

All mentioned methods improve denoising quality by modeling the imaging noise.
We, In contrast, are the first to show how blind spot denoising can be improved by including additional knowledge of the signal itself, namely the fact that it is diffraction-limited and oversampled.

While the blind spot architecture introduced in~\cite{laine2019high} is computationally cheaper than the masking scheme from \cite{krull2019noise2void,khademi2020self}, it is unfortunately incompatible with our setup (see Figure~\ref{fig:schema}).
Applying a convolution after a blind spot network would break the blind spot structure of the overall architecture.
We thus stick with the original masking scheme, which is architecture-independent and can directly be applied for end-to-end training.

\subsection{Other Unsupervised Denoising Approaches}
An important alternative route is based on the theoretical work known as \emph{Stein's unbiased risk estimator} (\SURE)~\cite{stein1981estimation}.
Given noisy observation, such as an image corrupted by additive Gaussian noise, 
Stein's 1981 theoretical work enables us to calculate the expected mean-squared error of an estimator that tries to predict the underlying signal without requiring access to the true signal.
The approach was put to use for conventional (non-deep-learning-based) denoising in~\cite{ramani2008monte} and later applied to derive a loss function for neural networks~\cite{metzler2018unsupervised}.
While it has been shown that the same principle can theoretically be applied for other noise models beyond additive Gaussian noise~\cite{raphan2007learning}, this has to our knowledge not yet been used to build a general unsupervised deep learning based denoiser.

In a very recent work called \DivNoising~\cite{prakash2020divnoising} unsupervised denoising was achieved by training a variational autoencoder (\VAE)~\cite{KingmaW13} as a generative model of the data.
Once trained, the \VAE can produce samples from an approximate posterior of clean images given a noisy input, allowing the authors to provide multiple diverse solutions or to combine them to a single estimate.

Like the previously discussed extensions of blind spot denoising~\cite{laine2019high,Krull:2020_PN2V,Prakash2019ppn2v,khademi2020self} all methods based on \SURE as well as \DivNoising rely on a known noise model or on estimating an approximation.
We, in contrast, do not model the noise distribution in any way (except assuming it is zero centered and applied at the pixel level) and achieve improved results.

A radically different path that does not rely on modeling the noise distribution was described by Ulyanov \etal~\cite{ulyanov2018deep}.
This technique, known as \emph{deep image prior}, trains a network using a fixed pattern of random inputs and the noisy image as a target.
If trained until convergence, the network will simply produce the noisy image as output.
However, by stopping the training early (at an adequate time) this setup can produce high-quality denoising results.
Like our self-supervised method, deep image prior does not require additional training data to be applied.
However, it is fundamentally different in that it is trained and applied separately for each image that is to be denoised, while our method can, once it is trained, be readily applied to previously unseen data.

\subsection{Concurrent Work on Denoising and Inverse Problems}
Kobayashi \etal~\cite{kobayashi2020image} developed a similar approach in parallel to ours.
They provide a mathematical framework on how inverse problems such as deconvolution can be tackled using a blind spot approach.
However, while we use a comparable setup, our perspective is quite different.
Instead of deconvolution, we focus on the benefits for the denoising task and show that the quality of the results on real data can be dramatically improved.

Yet another alternative approach was developed by Hendriksen \etal~\cite{hendriksen2020noise2inverse}.
However, this technique is limited to well-conditioned inverse problems like computer tomography reconstruction and is not directly applicable to the type of microscopy data we consider here.

\section{Methods}
\label{sec:methods}
In the following, we first describe our model of the image formation process, which is the foundation of our method, and then formally describe the denoising task.
Before finally describing our method for blind spot denoising with diffraction-limited data, we include a brief recap of the original \NoiseVoid method described in \cite{krull2019noise2void}.
\subsection{Image Formation}
\label{sec:imageFormation}
We think of the observed noisy image $\img$ recorded by the microscope, as being created in a two-stage process.
Light originates from the excited fluorophores in the sample.
We will refer to the unknown distribution of excited fluorophores as the \emph{phantom image} and denote it as $\latent$.
The phantom image is mapped through the optics of the microscope to form a distorted image $\sig$ on the detector, which we will refer to as \emph{signal}.
We assume the signal is the result of a convolution $\sig = \latent * \psf$ between the phantom image $\latent$ and a known \PSF $\psf$~\cite{richardson1972bayesian}. 

Finally, the signal is subject to different forms of imaging noise, resulting in the noisy observation $\img$.
We think of $\img$ as being drawn from a distribution $\img \sim \pnm{\img|\sig}$, which we call the \emph{noise model}.
Assuming that (given a signal $\sig$) the noise is occurring independently for each pixel, we can factorize the noise model as
\begin{equation}
    \pnm{\img|\sig} = \prod_i^N \pnm{\imgp_i, \sigp_i},
\end{equation}
where $\pnm{\imgp_i, \sigp_i}$ is the unknown probability distribution, describing how likely it is to measure the noisy value $\imgp_i$ at pixel $i$ given an underlying signal $\sigp_i$.
Note that such a noise model that factorizes over pixels can describe the most dominant sources of noise in fluorescent microscopy, the Poisson shot noise and readout noise~\cite{foi2008practical,zhang2019poisson}.
Here, the particular shape of the noise model does not have to be known. The only additional assumption we make (following the original \NoiseVoid~\cite{krull2019noise2void}) is that the added noise is centered around zero, that is the expected value of the noisy observations at a pixel is equal to the signal $\E{\imgp_i}{ \pnm{\imgp_i, \sigp_i}}= \sigp_i$. 

\subsection{Denoising Task}
\label{sec:denoisingTask}
Given an observed noisy image $\img$, the denoising task as we consider it in this paper is to find a suitable estimate $\sige \approx \sig$.
Note that this is different from the deconvolution task, attempting to find an estimate $\latente \approx \latent$ for the original phantom image.

\subsection{Blind Spot Denoising Recap}
\label{sec:bsdRecap}
In the originally proposed \NoiseVoid, the network is seen as implementing a function $\sigpe_i = f(\recf_i;\pars)$, that predicts an estimate for each pixel's signal $\sigpe_i$ from its surrounding patch $\recf_i$, which includes the noisy pixel values in a neighborhood around the pixel $i$ but excludes the value $\imgp_i$ at the pixel itself. 
We use $\pars$ to denote the network parameters.

The authors of~\cite{krull2019noise2void} refer to $\recf_i$ as a \emph{blind spot receptive field}.
It allows us to train the network using unpaired noisy training images $x$, with the training loss computed as a sum over pixels comparing the predicted results directly to the corresponding values of the noisy observation
\begin{equation}
    \sum_{i}
    \left( 
    \sigpe_i - \imgp_i 
    \right)^2
    .
    \label{eq:loss}
\end{equation}
Note that the blind spot receptive field is necessary for this construction, as a standard network, in which each pixel prediction is also based on the value at the pixel itself would simply learn the identity transformation when trained using the same image as input and as target.

To implement a network with a blind spot receptive field \NoiseVoid uses a standard \UNet~\cite{ronneberger2015u} together with a masking scheme during training.
The loss is only computed for a randomly selected subset of pixels $\setRandPix$. 
These pixels are \emph{masked} in the input image, replacing their value with a random pixel value from a local neighborhood.
A network trained in this way acts as if it had a blind spot receptive field, enabling the network to denoise images once it has been trained on unpaired noisy observations.

\subsection{Blind Spot Denoising for Diffraction-Limited Data}
\label{sec:ourMethod}
While the self-supervised \NoiseVoid  method~\cite{krull2019noise2void} can be readily applied to the data $\img$ with the goal of directly producing an estimate $\sige \approx \sig$, this is a sub-optimal strategy in our setting.

Considering the above-described process of image formation,
we know that, since $\sig$ is the result of a convolution with a \PSF, high-frequencies must be drastically reduced or completely removed.
It is thus extremely unlikely that the true signal would include high-frequency features as they are \eg visible in the \NoiseVoid result in Figure~\ref{fig:schema}.
While a network might in principle learn this from data, we find that blind spot methods usually fail at this and produce high-frequency artifacts.

To avoid this problem, we propose to add a convolution with the \PSF after the \UNet (see Figure~\ref{fig:schema}).
When we now interpret the final output after the convolution as an estimate of the signal $\sige \approx \sig$, we can be sure that this output is consistent with our model of image formation and can \eg not contain unrealistic high-frequency artifacts.

In addition, we can view the direct output before the convolution as an estimate of the phantom image $\latente \approx \latent$, \ie an attempt at deconvolution.

To train our model using unpaired noisy data, we adhere to the same masking scheme and training loss (Eq.~\ref{eq:loss}) as in \NoiseVoid.
The only difference being that our signal is produced using the additional convolution, thus enforcing the adequate dampening of high-frequencies in the final denoising estimate.
\subsection{A Positivity Constraint for the Deconvolved Image}
\label{sec:posConstr}
Considering that the predicted deconvolved phantom image $\latente$  describes the distribution of excited fluorophores in our sample (see Section~\ref{sec:imageFormation}), we know that it cannot take negative values.
After all, a negative fluorophore concentration can never occur in a physical sample.

We propose to enforce this constraint using an additional loss component, linearly punishing negative values.
Together with the original \NoiseVoid loss our loss is computed as 
\begin{equation}
    \frac{1}{|\setRandPix|}
    \sum_{i \in \setRandPix}
    \left( 
    \sigpe_i - \imgp_i 
    \right)^2
    +
    \lambda
    \frac{1}{N}
    \sum_{i=1}^\numpix
    \max(0, -\latentpe_i)
    \label{eq:lossFull},
\end{equation}
where $\numpix$ is the number of pixels and $\lambda$ is a hyperparameter controlling the influence of the positivity constraint.
Note that the new positivity term can be evaluated at each pixel in the image, while the \NoiseVoid component can only be computed at the masked pixels.

\section{Experiments and Results}
\label{sec:experiments}
In the following, we evaluate the denoising performance of our method comparing it to various baselines.
Additionally, we investigate the effect of the positivity constraint (see Section~\ref{sec:posConstr}).
Finally, we describe an experiment on the role of the \PSF used for reconstruction.

\subsection{Datasets}
\label{sec:data}
\miniheadline{Fluorescence Microscopy Data with Real Noise}
We used 6 fluorescence microscopy datasets with real noise.

The \textit{Convallaria}~\cite{Krull:2020_PN2V,Prakash2019ppn2v} and
\textit{Mouse actin}~\cite{Krull:2020_PN2V,Prakash2019ppn2v}
 datasets each consist of a set of 100 noisy images of $1024 \times 1024$ pixels showing a static sample.
The \textit{Mouse skull nuclei}~\cite{Krull:2020_PN2V,Prakash2019ppn2v} consist of a set of 200 images of $512 \times 512$ pixels.
In all 3 datasets, the ground truth is derived by averaging all images.
We use all 5 images in each dataset for validation and the rest for training.
The authors of~\cite{Krull:2020_PN2V,Prakash2019ppn2v} define a region of each image that is to be used for testing, while the whole image can be used for training of self-supervised methods.
We adhere to this procedure.

We additionally use data from~\cite{zhou2020w2s}, which provides 3 channels with training and test sets each consisting of $80$ and $40$, respectively. 
We use 15\% of the training data for validation.
Images are $512 \times 512$ pixels in size.
Note that like~\cite{prakash2020divnoising} we use the raw data made available to us by the authors as the provided normalized data is not suitable for our purpose.
The dataset provides 5 different versions of each image with different levels of noise.
In this work, we use only the version with the minimum and maximum amount of noise. 
We will refer to them as \textit{W2S avg1} and \textit{W2S avg16} respectively, as they are created by averaging different numbers of raw images.

\miniheadline{Fluorescence Microscopy Data with Synthetic Noise}
Additionally, we use 2 fluorescence microscopy datasets from~\cite{buchholz2020denoiseg} and added synthetic noise.
We will refer to them as \textit{Mouse (DenoiSeg)} and \textit{Flywing (DenoiSeg)}.
While the original data contains almost no noise, we add pixel-wise Gaussian noise with standard deviation 20 and 70 for \textit{Mouse (DenoiSeg)} and \textit{Flywing (DenoiSeg)}, respectively.
Both datasets are split into a training, validation, and test fraction.
The \textit{Mouse} dataset, provides 908 images of $128 \times 128$ pixels for training, 160 images of the same size as a validation set, and 67 images of $256 \times 256$ as a test set.
The \textit{Flywing} dataset, provides  1428 images size $128 \times 128$ as a training set, 252 images for validation (same size), and also 42 images size  $512 \times 512$ as test set.
As our method does not require ground truth, we follow \cite{prakash2020divnoising} and add the test fraction to the training data in order to achieve a fair comparison.

\miniheadline{Synthetic Data}
While the above-mentioned datasets are highly realistic, we do not know the true \PSF that produced the images.
To investigate the effect of a mismatch between the true \PSF and the \PSF used in the training of our method, we used the clean rendered text data from the book \emph{The beetle}~\cite{marsh2004beetle} previously introduced in~\cite{prakash2020divnoising}, synthetically convolved it using a Gaussian \PSF with a standard deviation of 1 pixel width.
Finally, we added pixel-wise Gaussian noise with a standard deviation of 100.
The resulting data consists of 40800 small images of $128 \times 128$ pixels in size. We split off a validation fraction of 15\%. 
\subsection{Implementation Details and Training}
\label{sec:implementation}
Our implementation is based on the \emph{pytorch}  \NoiseVoid implementation from~\cite{Krull:2020_PN2V}.
We use the exact same network architecture, with the only difference being the added convolution with the \PSF at the end of the network.

In all our experiments, we use the same network parameters:
A 3-depth \UNet with 1 input channel and 64 channels in the first layer.
All networks were trained for 200 epochs, with 10 steps per epoch.
We set the initial learning rate to 0.001 and used Adam optimizer, batch size = 1, virtual batch size = 20, and patch size = 100.
We mask 3.125\% (the default) of pixels in each patch.
We use the positivity constraint with $\lambda=1$ (see Section~\ref{sec:posConstr}).

\subsection{Denoising Performance}
\label{sec:denoisingPerformance}
We report the results for all fluorescence microscopy datasets in Table~\ref{tab:results}.
The performance we can achieve in our denoising task is measured quantitatively by calculation of the average peak signal-to-noise ratio (\textbf{PSNR}).
Qualitative results can be found in Figure~\ref{fig:table}.

We run our method using a Gaussian \PSF with a standard deviation of 1 pixel width for all datasets.
Figure~\ref{fig:table} shows examples of denoising results on different datasets.
\figTable
\tablePSNR

To assess the denoising quality of our method we compare its results to various baselines. 
We compared our method to \NoiseVoid, noise model based self-supervised methods (\PNtoV~\cite{Krull:2020_PN2V}, \DivNoising~\cite{prakash2020divnoising}), as well as the well-known supervised \CARE~\cite{weigert2018content} approach.
While we run \NoiseVoid ourselves, the PSNR values for all other methods were taken from \cite{prakash2020divnoising}.

We created a simple additional baseline by convolving the \NoiseVoid result with the same \PSF used in our own method.
This baseline is referred to as \emph{N2V (conv.)}.

\subsection{Effect of the Positivity Constraint}
\label{sec:effectOfPosConstr}
Here we want to discuss the effect of the positivity constraint (see Section~\ref{sec:posConstr}) on the denoising and deconvolution results.
We compare our method without positivity constraint ($\lambda = 0$, see Eq.~\ref{eq:lossFull}) and with positivity constraint ($\lambda = 1$).
Choosing different values for $\lambda$ did not have a noticeable effect.

We find that the constraint does not provide a systematic advantage or disadvantage with respect to denoising quality (see Table~\ref{tab:results}).
In Figure~\ref{fig:deconv} we compare the results visually.
While it is difficult to make out any differences in the denoising results, we see a stunning visual improvement for the deconvolution result when the positivity constraint is used.
While the deconvolution result without positivity constraint contains various artifacts such as random repeating structures and grid patterns, these problems largely disappear when the positivity constraint is used.

We find it is an interesting observation that such different predicted phantom images can lead to virtually indistinguishable denoising results after convolution with the \PSF, demonstrating how ill-posed the unsupervised deconvolution problem really is. 
\figDeconv

\subsection{Effect of the Point Spread Function}
\label{sec:effectOfPSF}
Here we want to discuss an additional experiment on the role of the \PSF used in the reconstruction and the effect of a mismatch with respect to the \PSF that actually produced the data.

We use our synthetic \emph{The beetle} dataset (see Section~\ref{sec:data}) that has been convolved with a Gaussian \PSF with a standard deviation of $\sigma=1$ pixel width and was subject to Gaussian noise of standard deviation 100.
We train our method on this data using different Gaussian \PSFs with standard deviations between $\sigma=0$ and $\sigma=2$.
We used an active positivity constraint with $\lambda=$ 1.
The results of the experiment can be found in Figure~\ref{fig:psf}.

We find that the true \PSF of $\sigma=1$ gives the best results.
While lower values lead to increased artifacts, similar to those produced by \NoiseVoid, larger values lead to an overly smooth result.
\figPSF

\section{Discussion and Outlook}
\label{sec:Discussion}
Here, we have proposed a novel way of improving self-supervised denoising for microscopy, making use of the fact that images are typically diffraction-limited.
While our method can be easily applied, results are often on-par with more sophisticated second-generation self-supervised methods~\cite{Krull:2020_PN2V,prakash2020divnoising}.
We believe that the simplicity and general applicability of our method will facilitate fast and widespread use in fluorescence microscopy where oversampled and diffraction-limited data is the default.
While the standard deviation of the \PSF is currently a parameter that has to be set by the user, we believe that
future work can optimize it as a part of the training procedure.
This would provide the user with an \emph{de facto} parameter-free turn-key system that could readily be applied to unpaired noisy raw data and achieve results very close to supervised training.

In addition to providing a denoising result, our method outputs a deconvolved image as well.
Even though deconvolution is not the focus of this work, we find that including a positivity constraint in our loss enables us to predict visually plausible results.
However, the fact that dramatically different predicted deconvolved images give rise to virtually indistinguishable denoising results (see Figure~\ref{fig:deconv}) illustrates just how underconstrained the deconvolution task is.
Hence, further regularization might be required to achieve deconvolution results of optimal quality.
In concurrent work, Kobayashi \etal~\cite{kobayashi2020image} have generated deconvolution results in a similar fashion and achieved encouraging results in their evaluation.
We expect that future work will quantify to what degree the positivity constraint and other regularization terms can further improve self-supervised deconvolution methods.

We believe that the use of a convolution after the network output to account for diffraction-limited imaging will in the future be combined with noise model based techniques, such as the self-supervised~\cite{Krull:2020_PN2V,laine2019high} or with novel techniques like \DivNoising.
In the latter case, this might even enable us to produce diverse deconvolution results and allow us to tackle uncertainty introduced by the under-constrained nature of the deconvolution problem in a systematic way.

\subsubsection*{Code Availability.}
\label{sec:code}
Our code is available at \url{https://github.com/juglab/DecoNoising}.

\subsubsection*{Acknowledgments.}
\label{sec:acknowledgments}
We thank the Scientific Computing Facility at MPI-CBG
for giving us access to their HPC cluster.

\par\vfill\par

\clearpage
\bibliographystyle{splncs04}
\bibliography{refs}
\end{document}


Peer reviews:

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 6: Marginally above acceptance threshold
review: The paper addresses the problem of denoising of microscopy images and the fact that traditional methods as well as various recent supervised deep learning methods make assumptions about the noise statistics that may not hold. The authors advocate the use of self-supervised deep learning methods, as high-quality paired training data is often not available to properly train supervised methods. But they observe that self-supervised methods typically produce high-frequency artifacts and achieve inferior results compared to supervised methods. To remedy this, they propose to exploit the fact that the images are usually diffraction-limited, by adding a convolution with a point-spread function model to an existing self-supervised deep learning-based denoising method (Noise2Void) and training it accordingly. Experimental results on a range of microscopy images illustrate the potential of the proposed method.

This paper is well written and the presentation is easy to follow. While the idea is interesting, I am not convinced it is theoretically sound. As explained (in Section 3.3 and also in Section 3.4), the Noise2Void method estimates the image s. Since s=z*h (Section 3.1), this makes it a denoising method, not a deconvolution method, and that is indeed how the method was designed. Thus, simply processing the estimated s by convolution with an assumed PSF model h (Figure 1 and Section 3.4) is questionable. Of course, doing so will force Noise2Void to behave more like it, and you can claim to "view the direct output before the convolution as an estimate of the phantom image ... i.e. an attempt at deconvolution" and get some visually pleasing results, but that does not make the approach theoretically right. Rather, it seems a practical trick that apparently happens to work to some extent.

Other specific comments:

- Section 4.1: Synthetic data is generated using a Gaussian PSF and pixel-wise additive Gaussian noise, but that is not realistic. As the authors admit elsewhere (multiple times), the dominant sources of noise are Poisson photon noise and Gaussian readout noise (Sections 1 and 3.1).

- Section 4.2: "Our implementation is based on the pytorch Noise2Void implementation from [10]. We use the exact same network architecture, with the only difference being the added convolution with the PSF at the end of the network." This, combined with the above major concern, make both the theoretical and the practical contribution of the paper rather limited.

- In Section 2 many methods are discussed but the comparison in Figure 2 is limited to only N2V (and a variant). Are there really no available software implementations of other methods to compare with?

- Section 4.3: The only quantitative measure used is PSNR. It is tricky to make the entire quantitative comparison hinge on a single measure that is known to be questionable. It would be good to also evaluate using other measures, such as SSIM.

- The authors claim "considerable visual improvements" (Figure 2) and even "stunning visual improvement (Section 4.4). These are subjective statements that in my opinion are not supported by the provided evidence.
title: Interesting practical method but questionable concepts and results

content: confidence: 4: The reviewer is confident but not absolutely certain that the evaluation is correct
rating: 6: Marginally above acceptance threshold
review: In this work, the authors present an extension to the Noise2Void denoising framework that incorporates convolution with a point spread function in order to better approximate the image formation process in microscopy.

### Quality
The premise of the work is very solid and represents a shift towards making denoising approaches specific to bioimaging data rather than just the direct translation of computer vision techniques originally formulated for e.g. photographs, video data. This is an important conceptual advancement in the field.

The performance of the new method is assessed in comparison to a selection of other denoising frameworks, and when measured by the PSNR metric is shown to out-perform comparable self-supervised methods. However, I find the general reliance on the PSNR as a performance assessment to be problematic, as this does not take into account the structural content of the images post-denoising. For example, in Fig. 2 there appears to be some structural discrepancies in the ‘Flywing’ data. While the ‘N2V (conv.)’ image has a lower PSNR than ‘ours’, the visual agreement between the N2V (conv.) and ground truth data appears better than that between ours and ground truth. As a sanity check for myself, I thresholded and skeletonised these images and while the N2V (conv.) and ground truth skeletons matched well, the ‘ours’ skeleton deviated substantially at the central junction. Apologies if this seems facetious, but I think it underlines the necessity for another measure of performance, especially as the ultimate goal of denoising microscopy images is to produce a better baseline from which quantitative measurements of structure can be made (rather than just a visually pleasing image).
I would suggest that the authors remove the phrase ‘stunning visual improvement’ (line 443) as this is rather subjective – for example, using the positivity constraint in the mouse actin deconvolution does not improve the prevalence of patterned noise (which can be seen if the images are Fourier-transformed).

Section 4.5, wherein the effects of the PSF size are investigated, seems a little abrupt. Although the PSF size parameter is clearly critical for the performance of the method, this section would have benefitted from additional discussion of e.g. a non-uniform PSF throughout the image or tolerance to the PSF deviating from a Gaussian function, as these are both relevant considerations in real-life microscopy applications.

### Clarity
The paper is overall incredibly clear and I managed to understand the majority of what was written on my first pass (in contrast to my general experience reading papers in this field).

The repeated use of the phrase ‘diffraction-limited’ is somewhat misleading and may even be doing the work a disservice, This phrase is normally used in the context of referring to conventional widefield or confocal fluorescence imaging data; however there is no reason that the application of this method is limited to this regime. For example, given that the condition of a point spread function whose spatial distribution can be approximated by a Nyquist-sampled Gaussian distribution of known width, this approach could be readily applied to some super-resolution data such as STED images. For this reason, the authors may wish to reconsider the use of the phrase ‘diffraction-limited’ although this is just a suggestion.

### Originality
The work described here is a very similar concept to that described in [Kobayashi et al (2020)](https://arxiv.org/abs/2006.06156). However the authors acknowledge this work in their discussion and given that there was less than one month between the submission of the work by Kobayashi et al and the BIC submission deadline I do not see this as shortcoming in originality (rather, unfortunate timing). Setting aside the paper by Kobayashi et al, this paper displays interesting conceptual novelty. In comparison to the paper by Kobayashi et al, this work (in my opinion) is much better focused toward the application of fluorescence microscopy and is reported in such a way that I feel it is more likely that a microscopist would preferentially use the method presented here. 

### Significance
The overall conceptual significance – integrating knowledge about the image formation process into the denoising method – is high, as I mentioned above. I am still not entirely convinced, however, that there is a *significant* increase in performance, as the PSNR values represent fairly marginal gains over to Noise2Void (alongside my above concerns regarding structural fidelity).

### Pros
* The paper is very well written, explained, and presented
* The theoretical benefits of the approach are substantial, and again the explanation of these is well-integrated into the paper
* The discussion of the paper shows that this work is a starting point and that the authors have thought about concrete ways to extend and improve it going forward. 

### Cons
* The authors have not convinced me from a quantitative point of view that the results are superior to existing self-supervised methods.
title: Theoretically and conceptually very strong, but I have concerns regarding the performance