Results in Optics 1 (2020) 100019

Contents lists available at ScienceDirect

Results in Optics
journal homepage: www.elsevier.com/locate/rio

Single-molecule localization by voxel-wise regression using convolutional
neural network
Toshimitsu Aritake a,⇑, Hideitsu Hino b, Shigeyuki Namiki c, Daisuke Asanuma c, Kenzo Hirose c, Noboru Murata aaSchool of Advanced Science and Engineering, Waseda University, 3-4-1 Okubo, Shinjuku-ku, Tokyo 169-8555, JapanThe Institute of Statistical Mathematics, 10-3 Midori-cho, Tachikawa, Tokyo 190-8565, Japanc
Graduate School of Medicine, The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo 113-8643, Japanb

A R T I C L E

I N F O

Keywords:
3D single‐molecule localization microscopy
Multi‐focal plane microscopy
Convolutional neural network
Regression‐based method

A B S T R A C T
Single‐molecule localization microscopy is widely used in biological research for measuring the nanostructuresof samples smaller than the diffraction limit. In this paper, a novel method for regression of the coordinates ofmolecules for multifocal plane microscopy is presented. A regression problem for the target space is decomposed into regression problems for small subsets of the target space. Then, a deep neural network is used tosolve these problems. By decomposing the regression problem, a fully convolutional neural network can beused to solve the regression problems. The computation of the network is efﬁcient, and a simple andparameter‐free loss function can be used to train the network. The proposed algorithm is validated by both simulated and real data obtained by quad‐plane microscopy.

1. Introduction
Fluorescence microscopy is widely used in biological research toanalyze the structure of in vivo samples. However, due to the diffraction limit of light, the resolution of conventional ﬂuorescence microscopy is limited to approximately 200 nm laterally and 500 nm
axially. To overcome this diffraction limit, a number of super‐
resolution microscopy methods (see Schermelleh et al., 2019 and references therein), including single‐molecule localization microscopy(SMLM) (Betzig et al., 2006; Hess et al., 2006), have been proposed.
The fundamental problem in super‐resolution microscopy is to estimate the true molecular distribution from observed images. In SMLM,
only several molecules are activated at a time using photoactivatablemolecules; therefore, the positions of the activated molecules can be
accurately estimated by a localization algorithm, such as Gaussian ﬁtting. Then, by integrating the localization results of many frames into
one image, a high‐resolution image can be obtained.
However, the problem of SMLM is its slow imaging speed, and thereasons for this are twofold. First, because ﬂuorescent molecules aresparsely activated in a frame to accurately localize the molecules, along acquisition time is required to obtain a large number of frames
for the reconstruction of high‐resolution images. The second reason
for the slow imaging speed is that conventional localization methods,
such as Gaussian ﬁtting (Betzig et al., 2006; Hess et al., 2006), com-pressed sensing (Gu et al., 2014), and spline interpolation (Babcock
and Zhuang, 2017; Li et al., 2017), require a long computation time
to localize molecules, especially when the size of an image is large
or the super‐resolution scale factor is large. Therefore, a faster localization algorithm that can accurately estimate densely activated molecule
locations from observations is necessary to achieve interactive
imaging.
Recently, deep neural networks (DNNs) have received increased
attention and have been successfully used in a wide variety of applications, including various imaging problems (Lucas et al., 2018), due totheir high predictive performance. In the past several years, DNNs
have been also applied to SMLM problems, such as molecule localization (Zelger et al., 2018; Ouyang et al., 2018; Nehme et al., 2018; Boydet al., 2018; Zhang et al., 2018), colorization (Hershko et al., 2019),
and background estimation (Möckl et al., 2020). In particular, DNNs
achieved molecule localization with a higher accuracy and speed thanconventional methods.
Conventional approaches for molecule localization use hand‐
crafted model and estimate molecules locations by solving an optimization problem. The beneﬁt of this approach is that the trainingof the model is not required. However, it is difﬁcult to model various
factors that may affect to the observation process for an accurate estimation. In addition, the computational cost to solve an optimization
problem of molecule localization is usually high and therefore a long

⇑ Corresponding author.
E-mail address: toshimitsu.aritake@ruri.waseda.jp (T. Aritake).

https://doi.org/10.1016/j.rio.2020.100019
Received 31 July 2020; Revised 28 October 2020; Accepted 3 November 2020
2666-9501/© 2020 The Authors. Published by Elsevier B.V.
This is an open access article under the CC BY-NC-ND license (http://creativecommons.org/licenses/by-nc-nd/4.0/).

T. Aritake et al.

Results in Optics 1 (2020) 100019

as continuous values, the trained network can be used for any target
resolution.
However, training a regression‐based neural network is not as simple as training a voxel‐based neural network. Because the network
directly estimates the set of coordinates of molecules, a function isrequired that quantiﬁes the difference between the set of true coordinates and the set of estimated coordinates. DeepLoco regards molecule
distributions as weighted empirical distributions and uses theweighted sample maximum mean discrepancy (MMD) (Gretton
et al., 2012) as the difference between two sets of coordinates. However, the difference calculated by the weighted sample MMD depends
on the kernel function and its variance parameter. It is difﬁcult toselect an optimal kernel and its parameter, however, as the optimal
choice varies depending on the density of the molecules and the target
resolution. This makes training the network more difﬁcult than forvoxel‐based methods. Another problem of DeepLoco is that it only
accepts ﬁxed‐size input. DeepLoco uses convolutional layers such as
VGG (Simonyan and Zisserman, 2015) to extract the features of agiven image and then uses the extracted features to estimate the coordinates of molecules and the conﬁdence by stacked residual blocks
used in a residual neural network (ResNet) (He et al., 2016). A residual
block is fully connected; therefore, the size of a feature must be ﬁxed
for the regression.
In this work, we propose a novel method to regress molecule locations by a DNN. The basic principle of our method is to decompose a
regression of multiple molecules into voxel‐wise regressions of a single
molecule. The beneﬁt of the decomposition is that training the network becomes much simpler by considering regression problems for
only one molecule. The architecture of the proposed network is simple,
and commonly used parameter‐free loss functions can be used to train
the network. An outline of our proposed method is presented in Fig. 1.
In our proposed method, a regression problem is decomposed into
two subproblems: the classiﬁcation of voxels and the regression ofmolecular coordinates of the voxels. We assume that at most one molecule exists in a voxel in the observed image. Then, we solve two subproblems under this assumption. The ﬁrst subproblem is theclassiﬁcation of voxels to distinguish voxels in an input image that contain a molecule from voxels that do not contain a molecule. This problem can be solved by the same method used in voxel‐based methods.
Here, the resolution of a voxel does not need to be enhanced by upsampling; therefore, the computational cost to solve this classiﬁcation
problem is much smaller than that of molecule localization by voxel‐
based methods. The second subproblem is the regression of the molecule coordinate in a voxel. Because we assume that only one molecule
exists in a voxel, we only need to estimate the coordinate of one molecule in the voxel by a regression method.
By decomposing the regression problem into two subproblems, theprobability of containing a molecule and the coordinate of the molecule in the voxel are estimated for each voxel. Because ﬂuorescenceis spread around the molecule, and it is only necessary to localize at
most one molecule in a voxel, we use a CNN to estimate the probability

computation time is required to process a large number of images. On
the other hand, DNN based methods use a ﬂexible model and insteadof hand‐crafting the model, the model is trained by a large amount ofdata for an accurate molecule localization. Also, although the training
a neural network takes several hours to several days, the trained network can estimate the molecules locations efﬁciently by a combinationof simple calculations.
The training of a neural network relies on the availability of a large
variety and amount of labeled data, which are pairs consisting of anobservation and a target molecule distribution. However, the true
molecule distribution of a real observation is generally unknown;
therefore, most SMLM algorithms use a reasonable generative model
to simulate observations from artiﬁcial molecule distributions. A pairconsisting of an artiﬁcial molecule distribution and an observation
generated from the distribution is used to train a neural network;
therefore, inﬁnite training data can be generated to train a network.
There are two main types of methods to localize molecules by
DNNs: voxel‐based methods and regression‐based methods. In voxel‐
based methods, such as Nehme et al. (2018, 2019) and Aritake et al.
(2020), a convolutional neural network (CNN) is used to localize themolecules. The target three‐dimensional (3D) space is split into voxels
at a predeﬁned target resolution. Then, molecules are localized by classifying the voxels into voxels containing a molecule and voxels notcontaining a molecule. These methods treat molecule localization asa classiﬁcation problem, and the output of the network is the probability of containing a molecule for each voxel. Because the ﬂuorescence ofa molecule spreads around the molecule, a CNN is used as a classiﬁer
to localize the molecules for each voxel. Because the target resolution
is higher than the resolution of the input images, the network contains
upsampling layers to enhance the resolution of the latter. To train the
network, the true molecule distribution is also voxelized and compared with the estimated molecule distribution. To compare distributions, the sum of the binary cross‐entropy (BCE) for each voxel isused as the objective value to train the network.
The drawback of voxel‐based methods is their slow computational
speed. When the size of the input image or the target resolution
increases, the number of intermediate representations becomes extremely large. Because the convolutional layers used in CNNs involve linear operations, the computation time largely depends on the numberof intermediate representations. Another shortcoming of voxel‐based
methods is that the architecture of the network must be modiﬁedand trained each time the target resolution is changed.
Unlike voxel‐based methods, DeepLoco (Boyd et al., 2018) is aregression‐based method that directly estimates the coordinates ofmolecules in the target 3D space. DeepLoco consists of feature extraction layers and regression layers. An important difference from voxel‐
based methods is that the network does not use upsampling layers forfeature extraction. Therefore, the number of intermediate representations does not signiﬁcantly increase due to the input image, and thecomputational cost is much lower than that of voxel‐based methods.
In addition, because the locations of molecules are directly estimatedFig. 1. Outline of proposed method for regressing molecule coordinates.
2

Results in Optics 1 (2020) 100019

T. Aritake et al.

Ω ¼ fði; j; kÞji ∈ f1; 2; . . . n1 g ^ j ∈ f1; 2; . . . n2 g ^ k ∈ f1; 2; . . . n3 gg. We
assume that all voxels have the same size and that the size of the voxels
is s1  s2  s3 , where sd ¼ Sd =nd ðd ¼ 1; 2; 3Þ. In addition, the coordinate of the center of the voxel Φijk is written as
xijk ¼ ðxijk;1 ; xijk;2 ; xijk;3 Þ ∈ R3 ; therefore, Φijk ¼ ½xijk;1  s1 =2; xijk;1 þ s1 =2½xijk;2  s2 =2; xijk;2 þ s2 =2  ½xijk;3  s3 =2; xijk;3 þ s3 =2.
The ﬂuorescence intensity at each voxel is determined by the coordinate of observation xijk , the coordinates of molecules in the target
space, the weights of the molecules, and the PSF, which depends on
the optical layout of the microscope. Let y ∈ Rnþ1 n2 n3 be an observed
image and zm ∈ Φ ðm ¼ 1; 2; . . . ; MÞ be the coordinate of a molecule,where M is the number of molecules in the target space Φ. The weightsthat control the peak ﬂuorescence intensity of a molecule are represented as wm ∈ ½0; 1 ⊂ R ðm ¼ 1; 2; . . . ; MÞ. Then, the ﬂuorescenceintensity is determined by these values and the PSF. The PSF is represented by f, and the ﬂuorescence intensity at xijk is formulated as

and coordinate for each voxel. In practice, we can use a single CNN tosolve both the classiﬁcation and regression subproblems simultaneously. In addition, we can use a simple loss function to calculate theobjective value for training the neural network. In this work, we use
the sum of the BCE and ‘1 distance as the loss function for each voxel.
These functions are parameter‐free and as simple as the objective function in voxel‐based methods; Therefore, training the network becomesas simple as in voxel‐based methods. However, our method does not
require the target resolution parameter and maintains the computational efﬁciency of regression‐based methods.
In this work, we apply our proposed method to an image obtainedby multifocal plane microscopy (MUM). Like other localization methods, we use a point spread function (PSF) that approximates the imaging process of MUM to generate artiﬁcial data, and we use the artiﬁcial
data to train our model. Whereas the lateral drift of MUM reduces theaccuracy of localization algorithms, our algorithm is robust to lateraldrift by being trained on artiﬁcial data containing lateral drift.
The contribution of this study are summarized as follows. We propose a novel method to regress the coordinates of molecules in the target space using a CNN. The key principle of our method is todecompose a regression problem into classiﬁcation and regression
problems for each voxel in an observed image. The proposed networkis also fully convolutional, and images of any size can be processed by
the same network. The comparison with other types of DNN based
method is summarize in the Table 1. The source code to train and test
our proposed model is available at https://github.com/t-aritake/voxel-wise-regression-SMLM.
The remainder of this paper is organized as follows. In Section 2,
we describe the generative model and PSF used in our work. In Section 3, our proposed method, which consists of classiﬁcation andregression problems, is formulated, and the architecture of the neuralnetwork to solve these problems is presented. Then, in Section 4, we
validate our method by applying it to both artiﬁcial and real datasets.
In Section 5, we conclude the paper.

y ijk ¼ f ðxijk ; fðzm ; wm ÞgM
m¼1 Þ:

The ﬂuorescence intensities of the molecules are disjoint; therefore,y ijk can be written as the linear combination of the ﬂuorescence of eachmolecule asM

y ijk ≈ ∑ wm f ðxijk ; zm Þ:In addition, the observation contains shot noise and Gaussian noise;
therefore, the observation can be modeled asM

y ijk ¼ ∑ wm f ðxijk ; zm Þ þ ɛ ijk

2.2. Point spread function (PSF)
In this study, although an observation model is not explicitly used
for molecule localization, it is used to generate artiﬁcial data to train amodel to localize molecules. However, true PSF is generally unknown;
therefore, we use an approximate PSF that models the imaging processof the microscope that is used to obtain the observations.
In this paper, we use quad‐plane microscopy as MUM, and fourcameras are used to observe ﬂuorescence at different depths of the target space Φ. Therefore, hereinafter, n3 ¼ 4 is assumed. The PSF ofquad‐plane microscopy is modeled by the following function:ðxz Þ2 þðxijk;2 zm;2 Þ2
f ðxijk ; zm Þ ¼ aðxijk;3  zhm;3 Þ exp  ijk;1 m;1
þ b;
ð4Þ
2
h

A set of images obtained by MUM represent the ﬂuorescence intensities in the target space. Let Φ ¼ ½0; S1   ½0; S2   ½0; S3  # R3 be a target 3D space that is observed by a microscope. Practically, theobserved image is discretized and ﬁnite‐dimensional; thus, we assume
that the target space Φ is discretized into n1  n2  n3 disjoint voxels,
where n1 ; n2 , and n3 are the number of bins along the x‐, y‐, and z‐axis,
respectively. We also assume that the elements of the observed image
are ﬂuorescence intensities observed at the center of the voxels. The
voxelsarerepresentedasΦijk ðði; j; kÞ ∈ ΩÞ,
where2ðrðxijk;3 zm;3 ÞÞ

where aðxÞ is the peak ﬂuorescence intensity, rðxÞ is the width of ﬂuorescence, and b is the background ﬂuorescence intensity. This PSF is
similar to the PSF used in Gu et al. (2014) for biplane microscopy
and models the distribution of ﬂuorescence as a Gaussian function.
The width of the PSF rðx3 Þ varies depending on the distance between
the molecule and the focal planes, and is described by the following
defocus curve:

Table 1
The comparison with other types of neural network methods.

rðxÞ ¼ r 0

DeepLoco

Proposedcontinuous
required
fast
ﬁxed

continuous
not required
fast
any

ð3Þ

where ɛ ijk consists of Poisson noise, which represents the shot noise,
and Gaussian noise, which represents the observation noise.

2.1. Observation model

discretenot required
slow
any

ði; j; kÞ ∈ Ω;

m¼1

In this section, we formulate the observation model and PSF used inthis work. In this work, we use MUM to observe the ﬂuorescence ofmolecules, and the ﬂuorescence intensities are observed at several
depths in the target space.

Voxel-based

ð2Þ

m¼1

2. Formulation

Type of the output
Parameter of the loss function
Computational Speed
Input size

ð1Þ

rﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃﬃ
 x 2
 x 3 x 4ﬃ
1þ
;
þA
þB
D
D
D

ð5Þ

where r 0 is the width of the PSF when a molecule is on the focal plane,
and D is the focus depth of the microscope. The peak aðxÞ of the PSF
depends on the width rðxÞ, and is modeled asaðxÞ ¼3a0
2πrðxÞ2:

ð6Þ

T. Aritake et al.

Results in Optics 1 (2020) 100019

of molecules zm ðm ¼ 1; 2; . . . ; MÞ are directly estimated by regression,^ denote the estimated coordinates of moleand z^m ðm ¼ 1; 2; . . . ; MÞ
^ is the estimated number of molecules. To estimate
cules, where M
the coordinates of molecules from an observed image y, we use a^

regression model g parameterized by θ; therefore, fz^m gM
m¼1 ¼ gðy; θÞ.
The accuracy of the regression model g depends on the parameter θ.
To train the model g for accurate regression, we solve the following
minimization problem:^

minimizeθ Lðfzm gM
zm gM
m¼1 ; f^m¼1 Þ;

ð8Þ

where L is the loss function to quantify the difference between the set oftrue coordinates of molecules fzm gM
m¼1 and the estimated coordinates of^ ¼ 1, simple loss functions such as the ‘1
molecules f^zm gM . If M ¼ M
m¼1

distance or ‘2 distance can be used as L. However, in general,^ > 1 and M – M,
^ and the sets fzm gM ; fz^m gM^ represent the disM; M
m¼1

Fig. 2. Width of observed ﬂuorescent beads and values of the defocus curve.
The observed width of the ﬂuorescent beads w at each depth x3 arerepresented by a circle. The red line represents the value of the defocus curve
to approximate the width with the parameters in Table 2. (For interpretationof the references to colour in this ﬁgure legend, the reader is referred to the
web version of this article.)

Table 2
Parameters of the point spread function.
Parameters

Value

a0
b
r0
D
A
B

5:00  107
0
1:33  102
3:02  102
7:37  104
6:27  103^

avoid the direct estimation of the molecule coordinates f^zm gM
m¼1
in the target 3D space, and allow a simple loss function to be used
to train the model.

Fig. 2 presents the width of the observed ﬂuorescent beads and thevalue of the defocus curve (5). The parameters of the PSF is decided as
Table 2 so that the PSF well approximates the width and peak intensityof the observed ﬂuorescence of the ﬂuorescent beads.
In addition, observation by MUM is affected by the lateral drift ofthe camera positions. The amount of lateral drift is identical for eachfocal plane, and we denote the amount of lateral drift of the kth focal
plane along the horizontal and vertical axis by Δk ¼ ðΔk1 ; Δk2 ; 0Þ ∈ R3 .
When lateral drift is considered, the observed ﬂuorescence at the coordinate xijk from a molecule located at zm is modeled as f ðxijk  Δk ; zm Þusing PSF. As the PSF is deﬁned in Eq. (4), the following equation
holds:
f ðxijk  Δk ; zm Þ ¼ f ðxijk ; zm þ Δk Þ:

m¼1

tribution of molecules in the target space. Therefore, the loss function
L must satisfy the following conditions to quantify the difference
between the two sets. First, L must be deﬁned between two sets.
Because different sets have a different number of elements, and the elements of a set are generally not ordered, the loss function L must
directly quantify the difference between the two distributions of molecules in the target space. Second, the loss function must be differentiable to train the model g by gradient‐based methods, which can beeasily applied for training.
By estimating the conﬁdence of regression with the coordinates,^ can be set to a sufﬁciently larger number than the true numberMof molecules M. However, the choice of the loss function is limited
due to the conditions mentioned above. For example, DeepLoco(Boyd et al., 2018) uses the weighted sample MMD (Gretton
et al., 2012) as the loss function. However, it is difﬁcult to identify
a loss function that is parameter‐free, easy to calculate, and easy tominimize for training a model. To overcome this difﬁculty, we

3.2. Proposed method
In this work, instead of solving a regression problem to estimate
the coordinates of multiple molecules in the entire target space, we
decompose the regression problem into a regression problem foreach voxel Φijk ðði; j; kÞ ∈ ΩÞ. Namely, a single regression problemof an entire image is decomposed into multiple regression problemsof small regions of the target space. Then, we assume that at mostone molecule is contained in a single voxel. This assumption is reasonable because ﬂuorescent molecules are sparsely activated; therefore, multiple molecules are rarely contained in a single voxel.
Under this assumption, a regression problem for a single molecule
is solved for each voxel. Therefore, a simple loss function can beused as the loss function to train the regression models for thevoxels.
In practice, if a molecule is not contained in a voxel, the regression
problem for the voxel cannot be solved, as the target variable for the
regression problem does not exist. Therefore, we solve the regression
problem for a voxel as the composition of two subproblems. The ﬁrst
subproblem is a binary classiﬁcation problem to distinguish whether avoxel contains a molecule, while the second subproblem is a regression
problem for the voxel. The second problem must be solved for voxelsthat contain a molecule.
Let cijk be a binary value that is a label of the binary classiﬁcation
subproblem, where
1 9m ∈ f1; 2; . . . Mg; zm ∈ Φijk ;cijk ¼0 otherwise:

ð7Þ

This equation implies that when the camera positions have lateraldrift Δk , the ﬂuorescence observed at xijk from the molecule at zm isidentical to the ﬂuorescence observed at xijk from the molecule atzm þ Δk . Therefore, the lateral drift of cameras must be considered
for accurate molecule localization. In practice, because the absoluteamount of drift cannot be estimated from observation, we consider
the relative amount of drift Δk 0 ðk ¼ 2; 3; 4Þ from the reference focal
plane k ¼ 1.
3. Method
3.1. Molecule localization by regression-based method
The problem of molecule localization is to estimate the coordinatesof molecules in the target space. In this work, we solve the molecule
localization problem as a regression problem. Namely, the coordinates
4Results in Optics 1 (2020) 100019

T. Aritake et al.

∑

We consider label cijk to be positive if cijk ¼ 1 and negative otherwise, and voxel Φijk is considered a positive or negative voxel according to its label. We also deﬁne a set of indices of positive voxels asΩþ ¼ fði; j; kÞjcijk ¼ 1g # Ω and a set of indices of negative voxels asΩ ¼ Ω n Ωþ . In addition, for a regression for a voxel, the coordinateof a molecule zm must be converted to the coordinate of a voxel.
Assuming zm ∈ Φijk , the coordinate of the molecule in voxel Φijk is thecoordinate relative to the size of the voxel, and is represented asζ ijk ¼ ðζ ijk;1 ; ζ ijk;2 ; ζ ijk;3 Þ, whereζijk;d ¼

zijk;d  xijk;d 1
þ ∈ ½0; 1
2
sd

ði;j;kÞ ∈ Ω

is much larger than the sum of the loss of positive voxels,∑

ði;j;kÞ ∈ ΩþAs evident in this equation, the coordinate zijk can be restored fromζ ijk and xijk . Note that the true coordinates of molecules ζ ijk are deﬁned
only for voxels that contain a molecule, namely, for all voxels
Φijk ðði; j; kÞ ∈ Ωþ Þ.
In summary, it is necessary to estimate the label of voxels in an
observed image y and the relative coordinates of the voxels containing
a molecule. We represent the estimated label and coordinate for voxel
Φijk by ^cijk and ^ζijk ¼ ð^ζ ijk;1 ; ^ζ ijk;2 ; ^ζ ijk;3 Þ, respectively. Then, a set of four^^
values ^ηijk ¼ ð^cijk ; ζ ijk;1 ; ζ ijk;2 ; ^ζ ijk;3 Þ is estimated for each voxel

minimizeθ

ð9Þ

where ^η ¼ ðηijk Þði;j;kÞ ∈ Ω ∈ R

is an array of estimated values. Here,instead of estimating the binary label, our model estimates the positive
label probability as ^cijk ∈ ½0; 1 for each voxel. It should be noted that
although the coordinates of molecules are estimated for all voxels,
the estimation is reliable only when the voxel contains a molecule.
Therefore, the probability ^cijk can also be seen as the conﬁdence ofthe estimated coordinate ^ζ ijk .
The model g is trained by minimizing the following loss function:Lðη; ^ηÞ ¼∑ LBCE ðcijk ; ^cijk Þ þði;j;kÞ ∈ Ω∑

ði;j;kÞ ∈ Ωþ

Lpos ðζijk ; ^ζijk Þ;∑ LBCE ðcijk ; ^cijk Þ þði;j;kÞ ∈ Ω0∑

ði;j;kÞ ∈ Ωþ

Lpos ðζ ijk ; ^ζ ijk Þ;

ð11Þ

where Ω0 ¼ Ωþ ∪fði; j; kÞ ∈ Ω jLBCE ðcijk ; ^cijk Þ ⩾ πg.
In this work, we use a CNN as the regression model gðy; θÞ, and thearchitecture of the model g is presented in Fig. 3. As the ﬁgure indicates, our model is a simple fully convolutional neural network, andthe label probability and coordinate are estimated for each voxel.
Because the ﬂuorescence of a molecule spreads around the molecule,the values of a voxel can be estimated from ﬂuorescence around thevoxel using convolution operations. In the ﬁrst two layers, 5  5 convolutions are used to extract features, while 3  3 convolutions areused in the remaining layers. Here, the width and height of intermediate representations are identical to those of the input image, and only
the number of channels is changed in each layer. Then, in each layer ofthe network except for the last layer, the rectiﬁed linear unit (ReLU) is
used as the activation function, followed by batch normalization (Ioffe
and Szegedy, 2015). In the last layer, a sigmoid function is used as theactivation function, as the class probability and the values of the relative coordinate are elements in ½0; 1. Finally, the array of size
n1  n2  16 is resized to n1  n2  4  4 so that the network outputs
four values for each voxel of the input image.
Although, we use the 8 layered simple CNN, shallower or deeper
networks can also be considered. If we use shallower network, computation become slightly faster and the training of the network becomeseasier. However, it is difﬁcult to extract appropriate features for theclassiﬁcation and the regression from the input image by shallow networks. On the other hand, deeper networks are able to extract more
complex features than shallow networks. Instead, the computation
become slightly slower, and the training of the network becomes difﬁcult and requires more training data. Also, more complex networks
such as U‐net used in Nehme et al. (2018) can also be used.
The beneﬁt of this model is that an input image of any width andheight can be processed because the model g is fully convolutional.
In addition, the number of elements of intermediate representations
is increased only by changing the number of channels; therefore, estimation by the network is computationally and memory efﬁcient.

Φijk ðði; j; kÞ ∈ ΩÞ by the model g as follows:n1 n2 44ðLBCE ðcijk ; ^cijk Þ þ Lpos ðζ ijk ; ^ζ ijk ÞÞ:

Therefore, a model trained by minimizing this loss function tends
to estimate the label probabilities as close to 0 even for positive voxels.
To address this problem, we use hard negative mining, which uses only
a part of the loss of negative voxels. This technique has been recently
used in learning a neural network for object detection (Liu et al.,2016), and allows the trained model to perform more accurate estimation for positive voxels. Let π be the pth largest loss of negative voxels
LBCE ðcijk ; ^cijk Þ ðði; j; kÞ ∈ Ω Þ, where the conﬁdence ^cijk is estimated by
the model g with the current parameters. Then, we solve the following
minimization problem to train g:ðd ¼ 1; 2; 3Þ:^η ¼ gðy; θÞ;

LBCE ðcijk ; ^cijk Þ;

ð10Þ

where ^cijk and ^ζ ijk are estimated by g as (9), and the parameters θ of g areis the BCE for the
optimized to minimize the loss function. Here, LBCE
iclassiﬁcation subproblem:
LBCE ðcijk ; ^cijk Þ ¼ cijk log ^cijk þ ð1  cijk Þ logð1  ^cijk Þ;and Lpos is the loss function for the regression subproblem in each voxel.
Because the true molecule coordinate ζ ijk is deﬁned only for voxels that
contain a molecule, this loss is calculated only for voxel
Φijk ðði; j; kÞ ∈ Ωþ Þ. Because we assume that only one molecule exists
in a voxel Φijk ðði; j; kÞ ∈ Ωþ Þ, we use the ‘1 loss function as follows:
Lpos ðζijk ; ^ζijk Þ ¼ kζijk  ^ζijk k1 :Training the model involves the class imbalance problem. Because
ﬂuorescent molecules are sparsely activated in the target space, thenumber of positive voxels is much smaller than the number of negative
voxels. Therefore, the sum of the loss of negative voxels,

Fig. 3. Architecture of a convolutional neural network. The activation function used in this model is the rectiﬁed linear unit (ReLU) and batch normalization
ζ ijk;2 ; ^ζ ijk;3 Þ ∈ ½0; 14 .
except for the last layer. In the last layer, sigmoid activation is used to predict the four values ð^cijk ; ^ζ ijk;1 ; ^5T. Aritake et al.

Results in Optics 1 (2020) 100019

The optimal parameter θ of the network is different according tothe optical layout of the microscope. Then, the user needs to train
the network so that the output of the network is close to the actualclass labels and the coordinates before applying to the real samples.
We use artiﬁcial data generated from the observation model describedin Section 2 to train the network g. We determine the number of molecules M and sample the coordinates of the molecules from a uniform
distribution of the target space Φ. These coordinates are converted
to the label and coordinates of the voxel; then, these values are used
as target values for the regressions. A set of observed images at eachfocal plane is calculated using the observation model. When calculating the sets of observed images, the relative lateral drift
Δk 0 ðk ¼ 2; 3; 4Þ is also randomly determined for each sample. By training a neural network using these data, the network becomes robust to
the lateral drift of camera positions.
We note that when the optical layout of the microscope is changed,
the network needs to be retrained. For example, when the type of optical lenses are changed, the PSF of the optical system is also changed.
Then the network needs to be retrained by the new training data generated from new observation model. However, when the optical system is only slightly changed, only small number of training data isrequired for ﬁne‐tuning the network.

Fig. 4. Optical layout of quad-plane microscope. The intermediate image isrelayed onto each camera via a pair of lenses (L1, f = 125.0 mm; L2,
f = 100.0 mm). TL, tube lens; M1, 1:1 beam-splitter mirror. The inset displays
the focusing planes of four cameras.

4. Experiments
In this section, we present the experimental results of localizationby the proposed method using both artiﬁcial images and real microscopy images. In the following experiments, the images observed by
the microscope described below were processed on an NVidia Tesla
V100 32 GB graphics processing unit (GPU) to localize molecules in
the images.

Table 3
Parameters for experiments with artiﬁcial data.

4.1. Optical layout of microscope
In this subsection, we present the experimental settings of the
microscope used in the experiments. The microscope settings were
the same as those used in prior work (Aritake et al., 2020). A multi‐
focus microscope equipped with four EM‐CCD cameras was constructed based on a commercial inverted microscope, as illustrated
in Fig. 4. A 640‐nm laser beam was focused on the back focal planeof a 100 oil immersion objective lens to illuminate an Alexa Fluor
647‐stained specimen at an excitation intensity of approximately
5 kW/cm2. The ﬂuorescence emitted from the specimen was collectedby the same objective lens. The ﬂuorescence image formed by theinternal tube lens of the inverted microscope was relayed by an achromatic lens (f = 125.0 mm), split twice by 1:1 beam‐splitter mirrors,
and refocused onto the four cameras via achromatic lenses(f = 100.0 mm, Thorlabs). The axial positions of the achromatic lenses
in front of the cameras were adjusted so that the four planes were at
400‐nm intervals in the Z‐axis direction of the specimen corresponding
to the conjugate planes of the sensor surface of the respective cameras.
The difference in the ﬁeld of view of the cameras was corrected bycoordinate registration using afﬁne transformation, whose parameters
were determined by images of multiple ﬂuorescent beads captured on
different cameras.

Parameter

Value

Target space size
Resolution of a voxel
Size of images
Number of molecules
Variance of Gaussian noise
Rate parameter of Poisson noise

S1 ¼ S2 ¼ 12; 288 [nm], S3 ¼ 1; 600 [nm]s1 ¼ s2 ¼ 192 [nm], s3 ¼ 400 [nm]n1 ¼ n2 ¼ 64; n3 ¼ 4
M ¼ 50σ2 ¼ 9λ ¼ 101.0].
Inaddition,weused
randomlateraldrift
Δk 0 ¼ ðΔk1 0; Δk2 0; 0Þ ðk ¼ 2; 3; 4Þ for each observation. The observed
value was then calculated by (3). An example of the training data ispresented in Fig. 5. We used Adam (Kingma and Ba, 2015) as the optimizer, where the parameters were β1 ¼ 0:9 and β2 ¼ 0:99, the initial
learning rate was set to 1:0  103 , and the batch size was 50. The
epoch number of the optimization was 20, and the dataset was randomly shufﬂed at the end of each epoch. Our method does not require
adjustment of the learning rate based on the number of epochs.
To validate the accuracy of the trained network, we used artiﬁcial
test images that were generated in the same way as the training data.
In this experiment, we estimated that a molecule existed in a voxel Φijk
if the label probability ^cijk exceeded a threshold value τ, and we used
τ ¼ 0:5 or τ ¼ 0:9. Fig. 6 provides the localization results that were
estimated by the trained network. As the ﬁgure indicates, most of
the molecules in the target space were localized close to the true coordinates for both threshold values. Unlike the voxel‐based method
which localize the molecule at predeﬁned target resolution, the coordinates of molecules are localized as continuous values. Althoughsome molecules were not localized for τ ¼ 0:9, the localization results
were almost identical for both threshold values; thus, most molecules
were localized accurately.
Fig. 7(a) presents the Jaccard index of binary classiﬁcation for various molecule densities. We used 1,000 artiﬁcial images, and the number of activated molecules differed for each density. As seen in Fig. 7(a), the Jaccard index of binary classiﬁcation slightly decreased as thedensity of activated molecules increased. This demonstrates that the4.2. Experiments with artiﬁcial data
We used 100,000 artiﬁcial frames generated by the observation
model to train the network. Each frame consisted of four images thatapproximated quad‐plane microscopy observations. The parametersof the experiments are presented in Table 3. The coordinates of molecules zm ðm ¼ 1; 2; . . . MÞ were uniformly randomly sampled from thetarget space Φ, and the weights of the molecules wm ðm ¼ 1; 2; . . . MÞ
were also randomly sampled from a uniform distribution in [0.3,6Results in Optics 1 (2020) 100019

T. Aritake et al.

Fig. 5. (a) Example of artiﬁcial molecule distribution containing 50 moleculesin the target space. The true locations of molecules are represented by blue
crosses. A list of coordinates of the molecules was used to train the network.
(b) Observed image simulated from the artiﬁcial distribution. The size of theimage was 64  64  4.
Fig. 6. Examples of molecule distribution of test data and molecule distribution estimated by the trained network with the threshold values (a) τ ¼ 0:5
and (b) τ ¼ 0:9. Blue crosses represent the true molecule coordinates, whileorange plus symbols represent the estimated coordinates. (For interpretationof the references to colour in this ﬁgure legend, the reader is referred to the
web version of this article.)binary classiﬁcation became more difﬁcult as the density increased
due to the overlap of ﬂuorescence. In practice, as the density of themolecules increased, the number of false‐positive (FP) cases increased
more than the number of false‐negative (FN) cases because close molecules are difﬁcult to localize correctly. However, as Fig. 7(b) indicates,
the average distance from a localized molecule to its nearest moleculealong each axis was close to zero for all densities. Although the average error along each axis was slightly biased, this is possibly because
the artiﬁcial training data did not have enough variety to alleviate
the effect of lateral drifts in average. In addition, the errorbar of
Fig. 7(b) shows the standard deviation of the error and Fig. 8 presents
the distribution of the errors along each axis for different thresholding
value and molecule density. Here, we selected the distribution of theminimum and maximum density in Fig. 7(b). As seen in the ﬁgures,the variance of the distribution increased as the molecule density

increased and the errors along each axis followed a Poisson‐like distribution. These results suggest that even when voxels were not completely correctly classiﬁed, molecules could be localized fromadjacent voxels.
Table 4 presents a confusion matrix of the binary classiﬁcation ofvoxels for a threshold value of 0.5. The label probability estimatedby the trained network was binarized using the threshold value. We
used 500 artiﬁcial images for this experiment, and 50 molecules wereactivated for each image. Therefore, a confusion matrix of a total of7

T. Aritake et al.

Results in Optics 1 (2020) 100019

Fig. 7. Error of binary classiﬁcation and error of regression for various molecule densities. (a) Jaccard index of binary classiﬁcation; (b) average and standard
deviation of the error between the estimated coordinate and the nearest molecule.

Fig. 8. Histograms of errors of regression for different density. The variance of the distribution mainly depends on the density of the molecules. Especially theerrors along z-axis are Poisson distributed and the variance of the distribution becomes large when the molecule density is high.
8Results in Optics 1 (2020) 100019

T. Aritake et al.

Table 4
Confusion matrix of binary classiﬁcation by the trained network. The estimatedlabel was binarized using a threshold value of τ ¼ 0:5.
True label

Estimated label

Positive
Negative

Positive

Negative

22,8782,089

1,844
8,165,189

Table 5
False positive and false negative cases considering the adjacent voxels of false
positive and false negative voxels.

False Positive
False Negative

Adjacent1

Adjacent2

Non-adjacent

1,399

445
318

0
372

8,192,000 voxels is presented in Table 4. When the coordinate of amolecule was close to the boundary of a voxel, the network sometimes
failed to estimate the correct label probability of the voxel. Instead, thenetwork estimated that a molecule existed in the neighboring voxel
closest to the true molecule location.
Additional details of the FP and FN cases are listed in Table 5. Adjacent1 lists the number of FP and FN voxels for which an FN voxel andits adjacent FP voxel had a one‐to‐one correspondence. In this case, amolecule was localized from the adjacent FP voxel. Similarly, Adjacent2 for FP indicates the number of FP voxels that were adjacent to TP or
FN voxels. In this case, FP voxels could be adjacent to multiple FN or
TP voxels. In addition, Adjacent2 for FN presents the number of FN
voxels that were adjacent to voxels whose labels were estimated to
be positive. Then, Non‐adjacent for FP indicates the number of FP voxels that were not adjacent to TP or FN voxels, and Non‐adjacent for FN
indicates the number of FN voxels that were not adjacent to TP or FP
voxels. As illustrated in Table 5, there were no FP voxels that were notadjacent to voxels that contained a molecule. This result suggests that
because the existence probabilities of molecules were estimated fromthe intensity of molecules, FP voxels only appeared close to voxels that
contained a molecule.
The processing speed of the network is displayed in Fig. 9. The
computation time consisted of the image loading time, image transfer
time to a GPU, and computation time on the GPU. As indicated in theﬁgure, the computational speed of our proposed method was as fast asthat of DeepLoco and faster than that of the voxel‐based method, especially when large images were processed. Our method achieved 200
fps for the localization of molecules including data loading and datatransfer; thus, our method is readily applicable to the real‐time localization of molecules. In addition, images of various sizes can be processed by the same network.

Fig. 9. Comparison of computational speed of deep-learning-based methods
for image sizes of 16  16  4; 32  32  4, …, 256  256  4. The computation time consisted of the time to load the images, time to transfer them tographics processing units (GPUs), and time to predict the coordinates andconﬁdence.

Fig. 10. Example of a frame with observed microtubules. The observed image
was a 256  256  4 image, and the difference in the ﬁeld of view of thecameras was corrected by afﬁne transformation. In the above ﬁgure, theﬂuorescence from ﬂuorescent beads are masked to zero and images arenormalized for better visibility of ﬂuorescence from the microtubules.

4.3. Experiments with real data
In this subsection, we present the experimental results of real dataof observed microtubules using the microscope described in Section 4.1. Methanol‐ﬁxed COS7 cells were used for stochastic opticalreconstruction microscopy (STORM) imaging of tubulin molecules
expressed inside the cells, as described in a previous study(Cleveland and Sullivan, 1985). Images were acquired at 22 Hz with
20‐ms exposure. The observed images were preprocessed by an afﬁnetransformation so that the pixel‐level translations of the images were
removed. However, it should be noted that sub‐pixel‐level translation
remained in the observed images because the lateral drift of the camera positions affected the observation. The size of the voxels in theobserved images was the same as in Section 4.2. An example of a frame
is presented in Fig. 10. Note that the ﬂuorescence intensity from beads
that are used as markers are blighter than the ﬂuorescence from sam-ples; hence, the ﬂuorescence from beads are masked to zeros for better
visibility of ﬂuorescence from specimen. The dataset consisted of
30,000 frames, and each frame consisted of 256  256 images
obtained from four focal planes. Therefore, the size of each frame
was n1 ¼ n2 ¼ 256 and n3 ¼ 4. Each frame was processed independently to localize the molecules.
Fig. 11(a) presents the 3D coordinates of molecules estimated fromall frames. Here, we used the threshold value τ ¼ 0:9. Fig. 11(b) presents a high‐resolution image generated by discretizing the estimatedcoordinates. Here, we set the super‐resolution scale factor along eachaxis to 8 and the voxel size of the high‐resolution images to
24  24  50 [nm]. The difference in depth is represented by colors.
9T. Aritake et al.

Results in Optics 1 (2020) 100019

Therefore, our method is applicable to a wide variety of optical layouts
if a reasonable generative model is provided and if it can be assumed
that at most one molecule exists in a voxel. For example, 3D localization from a single image can also be performed by changing the opticallayout of the microscope and the layout of the output voxels from thelayout of the input image. We plan to study the application to othertypes of microscopy in future work. The overlapped layout of voxels
may also improve the classiﬁcation accuracy of the voxels.
CRediT authorship contribution statement
Toshimitsu Aritake: Conceptualization, Methodology, Software,
Validation, Formal analysis, Writing ‐ original draft, Visualization.
Hideitsu Hino: Formal analysis, Conceptualization. Shigeyuki
Namiki: Investigation. Daisuke Asanuma: Investigation. Kenzo Hirose: Supervision. Noboru Murata: Project administration.
Declaration of Competing Interest
The authors declare that they have no known competing ﬁnancial
interests or personal relationships that could have appeared to inﬂuence the work reported in this paper.
Acknowledgments
This work was partly supported by JSPS KAKENHI Grant Nos.
17H01793, 18H03291, and JST CREST Grant No. JPMJCR1761,
JPMJCR14D7.
References
Aritake, T., Hino, H., Namiki, S., Asanuma, D., Hirose, K., Murata, N., 2020. Fast androbust multiplane single molecule localization microscopy using deep neuralnetwork. arXiv:2001.01893.
Babcock, H.P., Zhuang, X., 2017. Analyzing single molecule localization microscopy
data using cubic splines. Scientiﬁc Rep. 7 (1), 552. https://doi.org/10.1038/
s41598-017-00622-w. URL:/https://pubmed.ncbi.nlm.nih.gov/28373678 https://www.ncbi.nlm.nih.gov/pmc/articles/PMC5428856/.
Betzig, E., Patterson, G.H., Sougrat, R., Lindwasser, O.W., Olenych, S., Bonifacino, J.S.,Davidson, M.W., Lippincott-Schwartz, J., Hess, H.F., 2006. Imaging intracellularﬂuorescent proteins at nanometer resolution 313(5793), 1642–1645.https://doi.
org/10.1126/science.1127344.
Boyd, N., Jonas, E., Babcock, H., Recht, B., 2018. DeepLoco: Fast 3D localization
microscopy using neural networks. bioRxiv.https://doi.org/10.1101/267096.
Cleveland, D.W., Sullivan, K.F., 1985. Molecular biology and genetics of tubulin. Annu.
Rev. Biochem. 54(1), 331–366. pMID: 3896122.https://doi.org/10.1146/annurev.
bi.54.070185.001555.
Gretton, A., Borgwardt, K.M., Rasch, M.J., Schölkopf, B., Smola, A., 2012. A kernel twosample test. J. Mach. Learn. Res. 13 (null), 723–773.
Gu, L., Sheng, Y., Chen, Y., Chang, H., Zhang, Y., Lv, P., Ji, W., Xu, T., 2014. Highdensity 3D single molecular analysis based on compressed sensing. Biophys. J. 106(11), 2443–2449. https://doi.org/10.1016/j.bpj.2014.04.021.
He, K., Zhang, X., Ren, S., Sun, J., 2016. Deep residual learning for image recognition.
In: 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). pp.
770–778.https://doi.org/10.1109/CVPR.2016.90.
Hershko, E., Weiss, L.E., Michaeli, T., Shechtman, Y., 2019. Multicolor localization
microscopy and point-spread-function engineering by deep learning. Opt. Express
27 (5), 6158–6183.
Hess, Samuel T., Girirajan, Thanu P.K., Mason, Michael D., 2006. Ultra-high resolution
imaging by ﬂuorescence photoactivation localization microscopy. Biophys. J. 91(11), 4258–4272. https://doi.org/10.1529/biophysj.106.091116.
Ioffe, S., Szegedy, C., 2015. Batch normalization: accelerating deep network training by
reducing internal covariate shift. In: Proceedings of the 32nd International
Conference on International Conference on Machine Learning – Volume 37,
ICML’15, JMLR.org. pp. 448–456.
Kingma, D.P., Ba, J., 2015. Adam: a method for stochastic optimization. In: 3rdInternational Conference on Learning Representations, ICLR 2015, San Diego, CA,
USA, May 7–9, 2015, Conference Track Proceedings.
Li, Y., Mund, M., Hoess, P., Matti, U., Nijmeijer, B., Sabinina, V.J., Ellenberg, J., Schoen,
I., Ries, J., 2017. Fast, robust and precise 3d localization for arbitrary point spreadfunctions.https://doi.org/10.1101/172643.
URL:/https://www.biorxiv.
org/content/early/2017/08/10/172643.
Liu, W., Anguelov, D., Erhan, D., Szegedy, C., Reed, S., Fu, C.-Y., Berg, A.C., 2016. SSD:
Single shot multibox detector. In: Leibe, B., Matas, J., Sebe, N., Welling, M. (Eds.),
Computer Vision – ECCV 2016. Springer International Publishing, Cham, pp. 21–37.

Fig. 11. Visualization of localization results for a threshold value of τ ¼ 0:9.
(a) Scatter plot of localized molecules; (b) high-resolution image generated
from localization results.

Fig. 11 reveals the tubular structure of the microtubules which varies
depending on the depth.

5. Conclusion
In this paper, we propose a novel deep learning model for regressing the coordinates of molecules from an observed image obtained by
MUM. The key principle of our method is to decompose a regression
problem into classiﬁcation and regression problems for each voxel inan observed image. Then, the localization problem of each voxel issolved as a combination of a binary classiﬁcation and regression problem for a single molecule. By this decomposition, a fully convolutional
neural network can be used to solve the regression problem, and training the network is also simpliﬁed. In addition, our proposed method isparameter‐free, computationally efﬁcient, and can process images ofany size by the same model. The experimental results demonstrate
the effectiveness of our proposed method.
In this study, we applied our method to images obtained by MUM.
Then, we used the voxels in the input image to decompose the regression problem. However, the layout of the voxels can be modiﬁed.
10Results in Optics 1 (2020) 100019

T. Aritake et al.
Lucas, A., Iliadis, M., Molina, R., Katsaggelos, A.K., 2018. Using deep neural networks
for inverse problems in imaging: Beyond analytical methods. IEEE Signal Process.
Mag. 35 (1), 20–36. https://doi.org/10.1109/MSP.2017.2760358.
Möckl, L., Roy, A.R., Petrov, P.N., Moerner, W.E., 2020. Accurate and rapid
background estimation in single-molecule localization microscopy using thedeep neural network bgnet 117(1), 60–67.https://doi.org/10.1073/pnas.
1916219117.
Nehme, E., Weiss, L.E., Michaeli, T., Shechtman, Y., 2018. Deep-storm: super-resolution
single-molecule microscopy by deep learning. Optica 5 (4), 458–464. https://doi.
org/10.1364/OPTICA.5.000458.
Nehme, E., Freedman, D., Gordon, R., Ferdman, B., Weiss, L.E., Alalouf, O., Orange, R.,
Michaeli, T., Shechtman, Y. DeepSTORM3D: dense three dimensional localization
microscopy and point spread function design by deep learning (jun).
arXiv:1906.09957.

Ouyang, W., Aristov, A., Lelek, M., Hao, X., Zimmer, C., 2018. Deep learning massively
accelerates super-resolution localization microscopy. Nat. Biotechnol. 36 (5),460–468. https://doi.org/10.1038/nbt.4106.
Schermelleh, L., Ferrand, A., Huser, T., Eggeling, C., Sauer, M., Biehlmaier, O.,
Drummen, G.P., 2019. Super-resolution microscopy demystiﬁed.https://doi.org/10.
1038/s41556-018-0251-8.
Simonyan, K., Zisserman, A., 2015. Very deep convolutional networks for large-scale
image recognition. In: International Conference on Learning Representations.
Zelger, P., Kaser, K., Rossboth, B., Velas, L., Schütz, G.J., Jesacher, A., 2018. Threedimensional localization microscopy using deep learning. Opt. Express 26 (25),33166–33179. https://doi.org/10.1364/OE.26.033166.
Zhang, P., Liu, S., Chaurasia, A., Ma, D., Mlodzianoski, M.J., Culurciello, E., Huang, F.
Analyzing complex single-molecule emission patterns with deep learning. Nat.
Methods 11, 913–916.https://doi.org/10.1038/s41592-018-0153-5.

11