# Deep Learning-Assisted Localisation of Nanoparticles in synthetically generated two-photon microscopy images

Rasmus Netterstrøm, Nikolay Kutuzov, Sune Darkner, Maurits Jørring

Pallesen, Martin Johannes Lauritzen, Kenny Erleben, François Lauze

► To cite this version:

Rasmus Netterstrøm, Nikolay Kutuzov, Sune Darkner, Maurits Jørring Pallesen, Martin Johannes Lauritzen, et al.. Deep Learning-Assisted Localisation of Nanoparticles in synthetically generated two-photon microscopy images. 2023. hal-04034919

HAL Id: hal-04034919

https://hal.science/hal-04034919

Preprint submitted on 17 Mar 2023

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
Deep Learning-Assisted Localisation of
Nanoparticles in synthetically generated
two-photon microscopy images

Rasmus Netterstrøm<fnref n="1" />*†, Nikolay Kutuzov<fnref n="2" />†, Sune Darkner<fnref n="1" />, Maurits Jørring Pallesen<fnref n="1" />, Martin Johannes Lauritzen<fnref n="2" />,<fnref n="3" />, Kenny Erleben<fnref n="1" /> and François Lauze<fnref n="1" />

**Abstract**

Tracking single molecules is instrumental for quantifying the trans-
port of molecules and nanoparticles in biological samples, e.g., in brain
drug delivery studies. Existing intensity-based localisation methods are
not developed for imaging with a scanning microscope, typically used
for in vivo imaging. Low signal-to-noise ratios, movement of molecules
out-of-focus, and high motion blur on images recorded with scan-
ning two-photon microscopy (2PM) in vivo pose a challenge to the
accurate localisation of molecules. Using data-driven models is chal-
lenging due to low data volumes, typical for in vivo experiments.
We developed a 2PM image simulator to supplement scarce train-
ing data. The simulator mimics realistic motion blur, background
fluorescence, and shot noise observed in vivo imaging. Training a

<footnote n="1">¹*Department of computer science, University of Copenhagen,
Universitetsparken, København, 2100, Denmark.</footnote>

<footnote n="2">²Department of Neuroscience, University of Copenhagen,
Blegdamsvej, København, 2200, Denmark.</footnote>

<footnote n="3">³Department of Clinical Neurophysiology, Rigshospitalet,
Blegdamsvej, København, 2100, Denmark.</footnote>

<footnote>*Corresponding author(s). E-mail(s): rane@di.ku.dk;</footnote>

<footnote>Contributing authors: nkutuzov@sund.ku.dk; darkner@di.ku.dk;</footnote>

<footnote>mauritsjpallesen@gmail.com; mlauritz@sund.ku.dk;</footnote>

<footnote>kenny@di.ku.dk; francois@di.ku.dk;</footnote>

<footnote>†These authors contributed equally to this work.</footnote>
2 *Deep learning-assisted localisation of nanoparticles*

data-driven model with simulated data improves localisation quality
in simulated images and shows why intensity-based methods fail.

**Keywords:** deep learning, single molecule localization, two-photon microscopy.

# 1 Introduction

Tracking single molecules or nanoparticles helps to solve a variety of biomedical problems, e.g., drug design and delivery [1], studying the regulation of gene expression [2], determining mechanisms of viral infections [3], and many others. Single-molecule (or particle) tracking (SMT) is becoming a popular tool to study mechanisms of transport at the nanoscale in living cells and tissues [4]. In vivo imaging, especially in scattering tissues like the brain, becomes an ultimate challenge for SMT due to (i) low photon budgets causing low signal-to-noise ratio on recorded images, (ii) motion artefacts caused by the animal's breathing, heartbeat, movement, and (iii) severe motion blur caused by the slowness of scanning microscopy typically used in vivo. These factors challenge the accurate localisation and tracking of molecules on images recorded in vivo, for example with two-photon microscopy (2PM; Fig. 4a).

Object localisation and detection is a core topic in computer vision. Feature-based detection, using SIFT [5], HoG [6], to cite the most classically used, were replaced by deep learning (DL) approaches, notably [7, 8] and many were built for specialised situations in medical imaging. On images of single molecules recorded with CCD or EMCCD cameras, localisation is achievable even for images with low signal-to-noise ratio or (and) moderate motion blur [9, 10]. The localisation algorithms, however, require writing down an expression for the distribution of fluorescence in recorded images, which may not be possible for imaging moving molecules with 2PM: random motion of molecules overlaid with the laser beam's motion (Fig. 4b) can make modelling the distribution of fluorescence on a recorded image prohibitively complex. Neural networks have the potential to approximate these complex distributions of fluorescence and return the desired position of the molecule. Deep learning (DL) has been predominantly used for the localisation of immobile molecules with wide-field microscopes equipped with a CCD or EMCCD camera [11–13] in various in vitro experiments but not yet for localisation of moving molecules with 2PM in living animals.

Experimental data recorded in vivo can be so scarce and expensive to collect that training of the neural networks is more likely to rely on artificial data. Simulating large training data sets, informed by well-known theories of microscope image formation, has been used to train deep networks to localise immobile objects on images recorded with a CCD or EMCCD camera [11, 12]. In vivo imaging in scattering tissues, e.g. in the brain, however, is typically done with scanning 2PM, which records images pixel-by-pixel, line-by-line (Fig. 4b),
as opposed to collecting all image pixels simultaneously with a CCD cam-
era. Here we show how to train a neural network to localise single simulated
quantum dots (QDs) and how the commonly used intensity-based detections
fails to localise the QD on pixel-by-pixel and line-by-line simulated images.
We used a combination of realistic simulated data, based on modelling diffu-
sion of QDs during exposure and realistic motion of the laser beam. Many DL
approaches have been used for the detection and localisation of objects, both
with natural images and medical imaging modalities too. We demonstrate how
a neural network used for object localisation and detection for natural images,
can accommodate the QD detection and localisation task by training it on
simulated data, incorporating effects of motion blur caused by very different
rates between diffusion motion and camera motion, and then allow localisation
of QDs on images sampled from a simulated test dataset.

2 Results

We started by developing a protocol to generate realistic synthetic 2PM images
for training the FRCNN model [14–16], later used for the localization of QDs.
By combining a simple Brownian-like motion of QDs with a deterministic
motion of the laser beam we reproduced motion blur characteristics for scan-
ning microscopy (Fig. 2). We experimentally measured point spread function
(PSF) dimensions and estimated fluorescence background from experimental
images ($N_{bkg}$ in Eq. 1) to mimic real images as close as possible. Finally, with
Poisson noise, we simulated images that looked very similar to images recorded
in the brain in vivo (Fig. 2).

In Figure 3, we highlight the primary issue of intensity-based detectors in
four images. The intensity-based detector, TrackPy [17], detects high-intensity
pixels even though the high-intensity pixels are not the trajectory centroid of
the QD. Training directly on the centroid coordinate enables FCNN to detect
the trajectory centroid of the QD with higher accuracy. Then we conducted
three experiments to demonstrate further the value of including the trajectory
centroid in the DL training pipeline. Each experiment had a different diffusion
coefficient that provided a different motion blur. The first experiment had
a diffusion coefficient of 0.75µm²/s, the second of 1µm²/s and the third of
1.75µm²/s. Besides the different diffusion coefficients, we tested the robustness
of each model by changing the epsilon-threshold (ε) of the precision-recall
curve; The epsilons were 1,2 and 3, corresponding to a distance of 70nm,
140nm and 210nm given a pixel size of 0.07µm.

2.0.1 Experiment 1 - 0.75 µm²/s

This experiment concentrates on localising the trajectory centroid of a QD when point sources of light closely resemble the microscope's PSF. That ensures the trajectory centroid is close to the high-intensity pixels, hence less motion blur. We choose a realistic diffusion coefficient value of 0.75µm²/s.

**Figure 1 (b)** shows a QD moving with a diffusion coefficient of 0.75µm²/s,
4 Deep learning-assisted localisation of nanoparticles

<figure><img src="image_2.png" /><figcaption>**Fig. 1** Four artificially simulated images (a-d) with diffusion coefficient going from 0 to 1.75µm²/s. In images b-d, the green circle is the starting point of the trajectory, and the cyan circle is the ending point. (a) show a QD that diffuses with a diffusion coefficient of 0µm²/s, showing the microscope PSF with noise. (b-d) show QD diffusing with increasing diffusion coefficient: 0.75, 1 and 1.75µm²/s.</figcaption></figure>

<table><caption>**Table 1** This table shows the AUC-score of all three experiments. In the first column, the QD diffuse with 0.75µm²/s, in the second column, the QD diffuse with 1µm²/s and in the third column, the QD diffuse with 1.75µm²/s. In all three columns, the FCNN model achieves the best AUC-score.</caption><thead><tr><th rowspan="2">D<br>nm</th><th colspan="3">0.75µm²/s</th><th colspan="3">1µm²/s</th><th colspan="3">1.75µm²/s</th></tr><tr><th>70</th><th>140</th><th>210</th><th>70</th><th>140</th><th>210</th><th>70</th><th>140</th><th>210</th></tr></thead><tbody><tr><td>TrackPy</td><td>0.06</td><td>0.42</td><td>0.73</td><td>0.04</td><td>0.30</td><td>0.61</td><td>0.02</td><td>0.19</td><td>0.47</td></tr><tr><td>FCNN</td><td><strong>0.48</strong></td><td><strong>0.91</strong></td><td><strong>0.99</strong></td><td><strong>0.41</strong></td><td><strong>0.87</strong></td><td><strong>0.98</strong></td><td><strong>0.26</strong></td><td><strong>0.72</strong></td><td><strong>0.92</strong></td></tr></tbody></table>

allowing less motion blur. **Table 1, column 1** demonstrates that TrackPy and FCNN localise the trajectory centroid for localisation within 210nm with an AUC score of 0.73 and 0.99.

### 2.0.2 Experiment 2 - 1 µm²/s

We increase the diffusion coefficient, thus allowing more motion blur. **Figure 1 (c)** illustrates how the trajectory steps get larger, and the high-intensity pixels diverge from the microscope PSF; the distance between the trajectory centroid and high-intensity pixels becomes unpredictable. **Table 1, column 2** tells the immediate effect of more motion blur in the image. The intensity-based detector TrackPy drops from 0.73 to 0.61 AUC score for localisation within 210nm, whereas FCNN only drops from an AUC score of 0.99 to 0.98 for localisation within 210nm.

### 2.0.3 Experiment 3 - 1.75 µm²/s

At last, we make the high-intensity pixels diverge from the microscope PSF as much as possible for a realistic diffusion coefficient value. We use the diffusion coefficient of 1.75µm²/s, i.e., ten times larger than the diffusion coefficient used in experiment one. **Figure 1 (d)** illustrates how increasing motion blur scatters the high-intensity pixels across a larger area and, in some cases, makes the high-intensity pixels interchangeable for noise. Again we see in **table 1, column three**, that TrackPy drops in AUC performance, going from 0.61 to
0.47, whereas FCNN keeps an AUC score above 0.90 for localisation within 210nm.

<figure><img src="image_3.png" /><figcaption>Fig. 2 Comparison of three artificially simulated images (a-c) with real experimental images of QDs in the brain (d-f), recorded with 2PM. Note how the simulated data reproduces the motion blur: panels a, b, and f, show elongated fluorescence intensity distributions in the vertical direction due to the movement of the QDs. Scale bars are 500 nm wide.</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>Fig. 3 Localization of QDs on simulated images (a-d) Blue and purple crosses show the localisation of QDs estimated with FCNN and TrackPy, respectively. Green curves show simulated trajectories; green crosses show trajectory centroids.</figcaption></figure>

## 3 Discussion

To tackle the problem of accurate localization of QD in pixel-by-pixel recorded images, we developed a simulator that generates a particle undergoing a Brownian motion recorded by a two-photon laser scanning microscope. Implementing the motion of the laser beam during pixel-by-pixel image scanning allows the reproduction of complex motion blur typically observed in experimental data. Note, that our simulator simplifies local transport in vivo and 2PM. First, we compute the trajectory of the QD by Brownian motions and second, we only generate images with one QD showing. To simplify the simulation, we neglected the excess and readout noises [9] introduced by the detector (photomultiplier tube; PMT) and assumed that the actual number of recorded photons is Poisson-distributed. We show that realistic simulated data, which incorporates the effect of motion blur, non-specific background fluorescence, and Poisson shot noise, improves the deep learning-based localization of QDs, seen in experiments 1 to 3. The primary result of our experiments demonstrates that the FCNN is robust against motion blur and keeps an AUC score
6 *Deep learning-assisted localisation of nanoparticles*

above 0.90, whereas TrackPy drops the AUC score to 0.47. But we only trained
one model. So for future implementations, we intend to make a five-fold cross-
validation. To our knowledge, this is the first simulator of training data for
2PM. It opens for better and faster trajectory recovery and analysis, an invalu-
able tool to understand how molecules move in complex, turbid, biological
tissues, e.g., in the brain.

4 Methods

## 4.1 Data description

We simulated a training dataset of 29838 images and three testing datasets of 2570, 2410 and 1870 images. We chose a training dataset size of 29838 to avoid overfitting, and we chose a test dataset size of 6850 because it is common practice to choose a test dataset that is 20% of the size of the training dataset. The following parameters reproduce in vivo experiment parameters: All images were 24 pixels tall and 25 pixels wide, trajectories of the QDs were simulated with a set of diffusion coefficients between 0.75µm²/s and 1.75µm²/s, the pixel size was 0.07µm, the pixel exposure time was 2µs and the retracing time was 1.1ms. To ensure single-molecule localisation images contained 1 QD.

## 4.2 Deep Learning Model

We use the open-access, and easy-to-use PyTorch implementation of FRCNN-
keypoint with a resNet backbone [18] pre-trained on the MS COCO dataset
[19].

4.3 Train Configuration

We only trained our model using 20 epochs because the model converged. Of
the 20 models, we chose the model with the highest validation accuracy. Due
to limited GPU memory, we chose a batch size of 6 images, a validation batch
size of 4 images, and stochastic gradient descent. We tuned the learning rate
to 0.001 [20].

4.4 Materials

We used a commercial two-photon scanning microscope (FVMPE-RS) built
by Olympus (Japan), equipped with a Ti:Sapphire laser and GaAsP detectors
(photomultiplier tubes; PMTs). We used commercial QTracker655 quantum
dots (QDs; ThermoFisher Scientific, USA), which we diluted 15 times in saline
before intravenous injection into anaesthetized mice.

4.5 Animals and surgical procedures

All protocols involving mice were approved by the Danish National Commit-
tee on Health Research Ethics, in accord with the guidelines of the European
<figure><img src="image_5.png" /><figcaption>**Fig. 4** Imaging single quantum dots in the brain of living mice with two-photon microscopy (2PM). a A simplified scheme of the experimental setup: 2PM images quantum dots (QDs) inside the brain of a living, anaesthetised mouse by sweeping the laser beam across the brain and collecting fluorescence emitted by the QDs. b Images are recorded by moving the laser beam (by rotating x- and y-scan mirrors shown in a) along straight lines (solid red). Between the lines, the beam is retraced (dashed red): this ensures the same scanning speed as the laser beam. c Dimensions of the point spread function (PSF; drawn to scale) of a 2PM inside a brain [21] and a scheme of brain extracellular spaces — a 3D network of channels between brain cells membranes — where QDs are moving. In 2PM, the PSF represents the square of the laser intensity near the focus and it is approximately Gaussian, with standard deviation along the z-axis being ≈6 times larger than along x- and y-axes.</figcaption></figure>

Council's Convention for the Protection of Vertebrate Animals used for Experimental and Other Scientific Purposes, and complied with the ARRIVE guidelines. We used four male wild type mice (C57bl/6j; Janvier-labs). We followed standard surgical procedures including tracheotomy, cauterization, and craniotomy, described in detail elsewhere [1]. QDs were injected intravenously and released in the brain parenchyma by ablating a small capillary with high laser power.

## 4.6 Simulating 2PM images of diffusing QDs.

### 4.6.1 Laser beam motion simulation

First, we calculated the trajectory of the laser beam during image acquisition (Fig. 4b) assuming that the laser beam moves along straight-line segments at the constant speed of 2 µm per pixel (typically used in in vivo experiments) until it reaches the end of a scanning line. Then the laser beam is retraced (no photons are recorded during this phase) to the beginning of the next line, which takes 1.1 ms. The process is repeated until the whole image is recorded. We estimated the position of the laser beam with the same time step as for diffusion trajectories (δt).
8 Deep learning-assisted localisation of nanoparticles

4.6.2 Random walk

Second, we simulated a random walk trajectory (independently for all three coordinates, xyz) of QDs during image acquisition by cumulatively adding normally-distributed random numbers N(0, σ²), with zero mean and variance σ² = 2Dδt, where D is the diffusion coefficient of the QDs and δ is a time step. To align the timestep of the simulator with the timestep δ, we needed to sample from two normal distributions. The first distribution uses a timestep of 2µm, whereas the second uses the retracing timestep of 1.1ms.

4.6.3 Point Spread Function

Third, we modelled the expected photon count in individual image pixels. Assuming that the square of the laser light intensity (two photons are absorbed simultaneously at 2PM) near the focus is approximately a 3D Gaussian [22] with standard deviations σₓᵧ in the x y-plane and σ_z (depth) along the optical axis (Fig. 4c), the expected number of photons is

$$
N = \frac{N_{qd}}{2\pi\sigma_{xy}^2} \exp\left(-\frac{(X-x)^2 + (Y-y)^2}{2\sigma_{xy}^2}\right) \cdot \frac{1}{\sqrt{2\pi\sigma_z^2}} \exp\left(-\frac{(Z-z)^2}{2\sigma_z^2}\right) + N_{bkg} \quad (1)
$$

with (X, Y, Z) the position of the QD, (x, y, z) the position of a laser beam, N<sub>qd</sub> is the total possible number of emitted photons by a QD in the focal plane in the direction of the objective, and N<sub>bkg</sub> is expected number of photons of the fluorescent background (autofluorescence). We used the value N<sub>bkg</sub> = 8.4, which was estimated by averaging the background fluorescence on images recorded in the brain. The value of N<sub>qd</sub> was adjusted to imitate the signal-to-noise ratio of the brain images. We used realistic, measured in vivo, parameters of the 2PM PSF: σ<sub>xy</sub> = 0.21 µm and σ<sub>z</sub> = 1.36 µm [21].

4.7 Localization of QDs

Several DL-based detectors have been proposed in recent years [23, 24], sum-
marised in a recent review [25]. Many of these detectors have been tailored to
detect complex objects. QDs are simple practically point sources of light but
they do not image as the microscope’s PSF because of the motion blur. Further-
more, images of QDs can be very noisy due to the high fluorescence background
or movement of the QD out of focus. For such problems, the Faster R-CNN-
Keypoint (FRCNN) [14–16], which predicts object bounding boxes along with
key points, has proven very reliable and has available implementations and
was adopted in this work.

4.8 Metrics

For training the model we have used classical detection metrics for the bound-
ing boxes and keypoints predicted by FRCNN. Detection speed in frames per
second is of no interest in our problem. Precision and recall [26] are based on intersection over union (IoU):

$$IoU(\hat{B}, B) = \frac{area(\hat{B} \cap B)}{area(\hat{B} \cup B)} \qquad (2)$$

with $\hat{B}$ is a predicted bounding box and $B$ the associated ground truth box. Determining true positives (TP), false positives (FP), true negatives (TN), false negatives (FN) for both bounding box and keypoints are computed according to the COCO standard [19, 27]. For evaluation, we needed another metric since an intensity-based detector such as TrackPy does not predict bounding boxes [17]. Therefore, we computed the area under the curve (AUC) from the precision and the recall. In addition, we modified the precision and the recall seen in [28] to use an L2 distance instead of an IoU constraint.

$$precision = \frac{tp}{tp + fp} \qquad (3)$$

$$recall = \frac{tp}{tp + fn} \qquad (4)$$

That means we define a prediction to TP if the distance between the predicted key point and the ground truth key point is below or equal to a threshold $\epsilon$. FP is defined as if the distance exceeds the threshold or the prediction is duplicated. At last, FN is defined as if the detector fails to predict. We then computed the precision and the recall using equation eqs. (3) and (4). Finally, we computed the area under the curve (AUC) using precision and recall. Precision tells the probability that a prediction is correct, whereas recall tells the probability of detecting all possible ground truths. The AUC combines the metric, e.g. If the AUC score is close to 1, the detector correctly predicts and detects all possible ground truths. If the AUC score is close to 0, the detector either poorly predicts, fails to predict or both.

# Declarations

## Funding

Not applicable

## Conflict of interest/Competing interests

Not conflict or competing interests

## Ethics approval

Not applicable
**Consent to participate**

Not applicable

**Consent for publication**

Not applicable

**Availability of data and materials**

We released the data as open-access at the time of publication.

**Code availability**

Most of the code in this project is custom-made and will be open-access at the time of publication. However, we are happy to share the code upon request.

**Authors' contributions**

The first and second co-first authors contributed equally to this work. They made the primary work of this paper. The third, fifth and sixth authors contributed with writing, supervision and concept development. The fourth author helped with the programming

**References**

[1] Kucharz, K., Kristensen, K., Johnsen, K.B., Lund, M.A., Lønstrup, M., Moos, T., Andresen, T.L., Lauritzen, M.J.: Post-capillary venules are the key locus for transcytosis-mediated brain delivery of therapeutic nanoparticles. Nat. Commun. **12**(1), 4121 (2021)

[2] Elf, J., Li, G.-W., Xie, X.S.: Probing transcription factor dynamics at the single-molecule level in a living cell. Science **316**(5828), 1191–1194 (2007)

[3] Liu, S.-L., Zhang, Z.-L., Tian, Z.-Q., Zhao, H.-S., Liu, H., Sun, E.-Z., Xiao, G.F., Zhang, W., Wang, H.-Z., Pang, D.-W.: Effectively and efficiently dissecting the infection of influenza virus by quantum-dot-based single-particle tracking. ACS Nano **6**(1), 141–150 (2012)

[4] Godin, A.G., Varela, J.A., Gao, Z., Danné, N., Dupuis, J.P., Lounis, B., Groc, L., Cognet, L.: Single-nanotube tracking reveals the nanoscale organization of the extracellular space in the live brain. Nat. Nanotechnol. **12**(3), 238–243 (2017)

[5] Lowe, D.G.: Distinctive image features from scale-invariant keypoints. Int. J. Comput. Vision **60**(2), 91–110 (2004). https://doi.org/10.1023/B:VISI.0000029664.99615.94
[6] Dalal, N., Triggs, B.: Histograms of oriented gradients for human detection. In: IEEE Computer Society Conference on Computer Vision and Pattern Recognition, vol. 1, pp. 886–893 (2005)

[7] Ren, S., He, K., Girshick, R., Sun, J.: Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Methods. IEEE Transaction on Pattern Analysis and Machine Understanding **39**(6), 1137–1149 (2017)

[8] Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You only look once: Unified, real-time object detection. CoRR abs/1506.02640 (2015) 1506.02640

[9] Mortensen, K.I., Churchman, L.S., Spudich, J.A., Flyvbjerg, H.: Optimized localization analysis for single-molecule tracking and super-resolution microscopy. Nat. Methods **7**(5), 377–381 (2010)

[10] Mortensen, K.I., Flyvbjerg, H., Pedersen, J.N.: Confined brownian motion tracked with motion blur: Estimating diffusion coefficient and size of confining space. Frontiers in Physics **8**, 601 (2021)

[11] Petrov, P.N., Shechtman, Y., Moerner, W.E.: Measurement-based estimation of global pupil functions in 3D localization microscopy. Opt. Express **25**(7), 7945–7959 (2017)

[12] Boyd, N., Jonas, E., Babcock, H., Recht, B.: DeepLoco: Fast 3D Localization Microscopy Using Neural Networks (2018)

[13] Nehme, E., Weiss, L.E., Michaeli, T., Shechtman, Y.: Deep-STORM: super-resolution single-molecule microscopy by deep learning. Optica, OPTICA **5**(4), 458–464 (2018)

[14] He, K., Gkioxari, G., Dollár, P., Girshick, R.B.: Mask R-CNN. CoRR abs/1703.06870 (2017) 1703.06870

[15] PyTorch: Keypoint R-CNN. https://pytorch.org/vision/stable/models/keypoint_rcnn.html Accessed 2023-02-22

[16] Paszke, A., Gross, S., Massa, F., Lerer, A., Bradbury, J., Chanan, G., Killeen, T., Lin, Z., Gimelshein, N., Antiga, L., Desmaison, A., Kopf, A., Yang, E., DeVito, Z., Raison, M., Tejani, A., Chilamkurthy, S., Steiner, B., Fang, L., Bai, J., Chintala, S.: Pytorch: An imperative style, high-performance deep learning library. In: Wallach, H., Larochelle, H., Beygelzimer, A., d Alch’ie Buc, F., Fox, E., Garnett, R. (eds.) Advances in Neural Information Processing Systems 32, pp. 8024–8035. Curran Associates, Inc., ??? (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf
[17] Allan, D.B., Caswell, T., Keim, N.C., van der Wel, C.M., Verweij, R.W.: Soft-matter/trackpy: Trackpy V0.5.0. https://doi.org/10.5281/zenodo.4682814. https://doi.org/10.5281/zenodo.4682814

[18] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016)

[19] Lin, T., Maire, M., Belongie, S.J., Bourdev, L.D., Girshick, R.B., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common Objects in Context, vol. V, pp. 740–755 (2014)

[20] Vision, P.: Pytorch Vision. https://github.com/pytorch/vision Accessed 2023-03-17

[21] Kutuzov, N., Flyvbjerg, H., Lauritzen, M.: Contributions of the glyco-calyx, endothelium, and extravascular compartment to the blood-brain barrier. PNAS 115(40), 9429–9438 (2018)

[22] Zipfel, W.R., Williams, R.M.E., Webb, W.W.: Nonlinear magic: multiphoton microscopy in the biosciences. Nature Biotechnology 21, 1369–1377 (2003)

[23] Redmon, J., Divvala, S.K., Girshick, R.B., Farhadi, A.: You Only Look Once: Unified, Real-Time Object Detection. In: Proceedings of the IEEE International Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016)

[24] Lin, T.-Y., Goyal, P., Girshick, R.B., He, K., Dollár, P.: Focal Loss for Dense Object Detection. In: International Conference on Computer Vision, pp. 2980–2988 (2017)

[25] Youzi, X., Zhiqiang, T., Jiachen, Y., Yinshu, Z., Liu, S., Shaoyi, D., Xuguang, L.: A review of object detection based on deep learning. Multimedia Tools and Applications 79(33-34), 23729–23791 (2020)

[26] Liu, L., Ouyang, W., Wang, X., Fieguth, P., Chen, J., Liu, X., Pietikäinen, M.: Deep Learning for Generic Object Detection: A Survey

[27] ppwwyyxx, rbg: pycocotools. GitHub (2018)

[28] Youzi, X., Zhiqiang, T., Jiachen, Y., Yinshu, Z., Liu, S., Shaoyi, D., Xuguang, L.: A review of object detection based on deep learning. Multimedia Tools and Applications 79(33-34), 23729–23791 (2020). Copyright - © Springer Science+Business Media, LLC, part of Springer Nature 2020; Last updated - 2020-08-22