# Inverse 3D Microscopy Rendering for Cell Shape Inference with Active Mesh

Sacha Ichbiah

Anshuman Sinha

Fabrice Delbary

Hervé Turlier

Collège de France, CNRS, Inserm, PSL University, Paris, France

hervé.turlier@cnrs.fr

## Abstract

Traditional methods for biological shape inference, such as deep learning (DL) and active contour models, face important limitations in 3D. DL approaches require large annotated datasets, which are often impractical to obtain, while active contour methods depend on carefully tuned heuristics for intensity attraction and shape regularization. We introduce deltaMic, a novel differentiable 3D renderer for fluorescence microscopy that formulates shape inference as an inverse problem. By leveraging differentiable convolutions, deltaMic simulates the image formation process, integrating a parameterized point spread function (PSF) with a triangle mesh-based representation of biological structures. Unlike DL- or contour-based segmentation, deltaMic directly optimizes both shape and optical parameters to align synthetic and real microscopy images, removing the need for large datasets or sample-specific fine-tuning. To ensure scalability, we implement a GPU-accelerated Fourier transform for triangle meshes along with narrow-band spectral filtering. We show that deltaMic accurately reconstructs cell geometries from both synthetic and diverse experimental 3D microscopy data, while remaining robust to noise and initialization. This establishes a new physics-informed framework for biophysical image analysis and inverse modeling.

## 1. Introduction

Fluorescence microscopy [51] is the most widely used technique for imaging biological structures. However, extracting quantitative information from 3D fluorescence images remains a major bottleneck, limiting the development of new biological shape analysis methods and models. In fluorescence microscopy, biological samples are labeled with fluorescent dyes (fluorophores) that bind to specific structures of interest—such as lipid membranes, cytoskeletal networks, or organelles—making them bright while keeping the background dark. Except for unavoidable camera noise and biological heterogeneities, 3D fluorescent images are typically sparse and composed of well-structured objects such as points, filaments, surfaces, or bulk solids.

Leveraging this inherent structure, tools based on deep-learning (DL) tailored for fluorescence imaging [5], particularly convolutional neural networks (CNNs) [13, 72], have enabled significant advances in automating challenging tasks such as image restoration [83], instance segmentation [74, 78], and feature encoding via self-supervised learning [44]. Despite these achievements, the expressivity of neural networks also presents challenges in 3D biological imaging: (1) Data requirements: 3D CNNs or newer architectures (e.g., Vision Transformers [19]) require large annotated datasets, which are difficult to produce and label accurately in 3D. While foundation models for biomedical image analysis show promise [54], commonly used DL models remain highly dependent on specific microscope imaging modalities, requiring extensive retraining when imaging conditions change. (2) Lack of interpretability and physical constraints: widely-used DL segmentation models [78] lack interpretability and rarely incorporate prior knowledge about optical image formation, limiting their scientific insight. (3) Geometric analysis: CNNs typically produce segmentation masks, which are suboptimal for measuring geometric features like curvatures or angles [34]. Additionally, timelapse microscopy requires object shape tracking [81], yet most segmentation models operate frame by frame, necessitating a separate shape-matching step as post-processing. (4) Risk of hallucinations: DL models can generate artifacts or structures that do not correspond to real biological features—an issue for scientific research and medical applications, where accuracy is critical.

In response to these challenges, a growing trend in microscopy image analysis favors methods that incorporate stronger prior knowledge of optical physics [3, 4, 48, 76]. In contrast to DL, active contour methods—also known as active snakes or meshes—directly evolve a shape representation to minimize an objective function aligned with image features [12, 20, 21, 38, 55, 75, 80]. Although being developed for decades, these methods have seldom incorporated the physical principles of fluorescence image formation, instead relying on user-defined regularization terms to enforce smoothness. While DL approaches recently sup-
<figure><img src="image_1.png" /><figcaption>Figure 1. Graphical summary: joint inference of cell geometry and optical PSF using our differentiable 3D fluorescence microscopy.</figcaption></figure>

planted them due to superior speed and automation, active contour methods retain valuable advantages: they can track objects over time and produce explicit shape representations (e.g., meshes, level sets) that ease geometric analysis and integrate naturally with downstream modeling frameworks.

Meanwhile, the emergence of libraries such as PyTorch [69] and JAX [26], which offer native automatic differentiation, has accelerated the adoption of differentiable models in computer vision (CV), notably in inverse rendering [41, 65] and mesh-based shape inference [77]. Although inverse rendering has proven effective in CV, it remains largely unexplored in fluorescence biological image analysis. Here, we introduce a novel paradigm that leverages the fundamental optics of fluorescence microscopy to approximate the 3D image formation process from biological object shapes. We present *deltaMic*, a differentiable model that renders 3D fluorescence microscopy images from surface meshes, which can be directly compared to real biological images using a weighted voxel-based $L^2$ norm. Implemented in PyTorch for automatic differentiation, our approach efficiently optimizes this loss by computing gradients with respect to both the biological object's shape and the microscope's optical properties. As a result, our method enables precise biological shape inference while simultaneously emulating the microscope's optics, eliminating the need for per-sample hyperparameter tuning required in traditional active contour methods.

**Main contributions:**

* We introduce a simplified model of 3D fluorescence microscopy generation, combining a mesh-based object representation with a parameterized point spread function (PSF).
* We implement a differentiable Fourier transform of triangle surface meshes parallelized on GPU.
* We demonstrate that our differentiable microscopy renderer can accurately infer 3D cellular shapes from both synthetic and real 3D microscopy images without requiring additional shape-regularization terms.
* We provide a PyTorch implementation on GitHub [35].

## 2. Related work

### 2.1. Fluorescence microscopy modeling

In fluorescence microscopy, a laser excites fluorescent dyes bound to biological structures at a specific wavelength, causing them to emit light at a longer wavelength. The emitted fluorescence passes through the optics and is captured by a CMOS camera, producing a 2D image. In modalities such as confocal or light-sheet microscopy, the focal plane is incrementally shifted to acquire a 3D volumetric image composed of optical sections [42]. However, resolution is fundamentally limited by the diffraction of light through the optical system [36]. When the imaging system's response is approximately translation-invariant in 3D, it can be described by a point spread function (PSF), which characterizes how a point source is imaged. Various physics-based PSF models have been proposed [2, 27, 29, 30, 45], mainly for deconvolution purposes [73]. The simplest way to determine a PSF is to image sub-micrometer fluorescent beads approximating point sources. The model is then fitted to these experimental images. In this work, we approximate the PSF using a basic Gaussian kernel for simplicity; however, the framework could also incorporate more accurate experimental or physics-based models [1].

Knowing the PSF also enables synthetic image generation, explored in several studies. These approaches either replicate the real image-formation process [18] or use texture synthesis [58, 84] or generative DL [23, 32] to produce realistic images. Such images are primarily used to evaluate image-analysis algorithms [70] or generate annotated datasets for DL training [61]. In [18], the fluorophore distribution is modeled as a Boolean mask convolved with a user-provided PSF and noise. We extend this approach by defining a fluorophore density over each mesh simplex rather than using discrete sources or a mask. This avoids densely populating biological meshes with millions of points and significantly reduces computation.
## 2.2. Instance segmentation of fluorescent images

Biological image segmentation has evolved in parallel with computer vision [59], from traditional methods such as thresholding, watershed [6, 7, 24], and active contours [12, 21, 38, 55], to graph-cut optimization [8], and more recently DL-based pipelines [13, 72]. In 2D, the most effective methods leverage large annotated datasets to train CNNs [14, 28, 68] or, more recently, vision Transformers [54]. These models predict instance masks robustly across samples and imaging modalities. However, extending this success to 3D remains challenging.

3D images (z-stacks) are acquired by capturing 2D slices at varying depths, introducing anisotropy along the z-axis that varies with imaging conditions, yielding heterogeneous datasets that hinder CNN training and generalization. Modern 3D images—ranging from 512³ voxels in confocal to 2048³ in light-sheet microscopy—also impose substantial memory constraints on GPUs. Labeling 3D data poses major challenges. It requires expert biological knowledge, strong spatial skills to maintain cross-plane consistency, and suitable annotation tools. The shift from 2D to 3D increases both the number of voxels and annotation complexity.

Classical contour-based segmentation techniques often minimize an energy functional $E(\Lambda, m)$ modeling the distance between a shape $\Lambda$ and image features $m$ [62], such as intensity gradients [57] or homogeneous regions [12]. Shape representations use level sets or meshes, and the optimal shape $\Lambda^*$ is found via gradient-based optimization. The negative gradient $\frac{\partial E(\Lambda, m)}{\partial \Lambda}$ acts as a force guiding the contour—hence the terms active snakes, contours [38, 55, 80], or meshes [21, 75]. These methods typically require user-defined regularization (e.g., penalization terms akin to tension or bending energies) to maintain smoothness and avoid artifacts.

Fluorescence images of slender structures like membranes are challenging for region- or edge-based energies, as such structures appear as thin, high-intensity features against a dark background. Alternatives use forces based on proximity to local intensity maxima [55, 79] or compare real and synthetic images generated from meshes [31, 64], where the synthetic image is produced by convolving a binary mesh mask with a known PSF. Our approach builds on this idea but introduces a more rigorous 3D rendering framework and direct variational formulation. We jointly optimize the mesh and PSF parameters in a differentiable pipeline, eliminating the need for explicit shape regularization while maintaining robust and physically grounded reconstructions.

## 2.3. Differentiable rendering

Rendering 2D or 3D geometrical shapes into rasterized images is a core problem in computer graphics. Recent advances have enabled differentiable rendering, allowing in-

verse problems where scene parameters (e.g., shapes, textures, materials) are learned from raster images [40, 50, 66]. Traditional rasterization is non-differentiable due to occlusions and discontinuities, prompting development of smoothing strategies [39, 46, 53, 71].

Our fluorescence microscopy emulator functions as a 3D differentiable renderer. Like in graphics, it relies on mathematically differentiable operations. Unlike graphics, the resulting image lives in 3D space. Image formation is smoothed by convolving the mesh with a PSF, similarly to smoothing in differentiable rasterization [53]. Since a triangle mesh may include thousands of vertices and a PSF hundreds of parameters, optimization demands efficient differentiation. Assuming all components are differentiable, reverse-mode differentiation (backpropagation) computes gradients via the chain rule. Growing demands for scalability have led to GPU-accelerated automatic differentiation libraries [26, 69]. We leverage PyTorch's backpropagation engine and implement our mesh Fourier Transform (FT) as a differentiable function within it.

<figure><img src="image_2.png" /><figcaption>Figure 2. *DeltaMic* pipeline overview: The microscopy renderer generates a 3D synthetic fluorescence image from a triangle mesh and a parameterized PSF emulating microscope optics. Gradients with respect to mesh and PSF parameters are computed via backpropagation, enabling their optimization to match real microscopy images.</figcaption></figure>

# 3. Fluorescent 3D image rendering

Images can be represented as intensity maps $I$ from $[0, 1]^3$ to $\mathbb{R}$ without loss of generality, as any non-cubic image can be linearly rescaled to fit within $[0, 1]^3$. Following subsections introduce approximated models for both: imaging system and the geometry of biological samples, that are then integrated to construct our differentiable renderer.

## 3.1. Translation-invariant rendering model

We assume that the PSF $h$ of the fluorescence microscope is translation-invariant. Under this assumption, a smooth image $I: [0, 1]^3 \to \mathbb{R}$ is obtained by convolving a spatial
<figure><img src="image_3.png" /><figcaption>Figure 3. Principle of 3D fluorescence microscopy rendering: A fluorophore distribution on the cell surface Λ is convolved with the system's PSF to produce a 3D fluorescence image.</figcaption></figure>

fluorophore density $u_Λ: [0, 1]^3 \to \mathbb{R}$ with the point spread function (PSF) kernel $h: [0, 1]^3 \to \mathbb{R}$:

$$I(\mathbf{x}) = (u_{\Lambda} * h)(\mathbf{x}) = \int_{[0,1]^3} u_{\Lambda}(\mathbf{p})h(\mathbf{x} - \mathbf{p})d^3\mathbf{p}. \quad (1)$$

To circumvent the direct computation of these integrals, we perform the convolution in Fourier space by applying an element-wise multiplication of the Fourier transform (FT) $\hat{u}_Λ$ and $\hat{h}$, yielding: $\hat{I} = \hat{u}_Λ \cdot \hat{h}$.

With this image formation model, one can start from a first guess $(I^0, h^0)$, and minimize the distance between the rendered image $I^0$ and a real microscopy image $I_{GT}$. Doing so allows to infer both the fluorophore density of the observed biological sample and the PSF of the imaging system: $(I^*, h^*)$. However, to learn meaningful shape representations, both of these functions have to be parameterized using prior knowledge on the shape considered, its topology, as well as on the PSF.

## 3.2. Point spread function model

The simplest PSF model is a Gaussian kernel, which is fully determined by its covariance matrix $\Sigma \in \mathbb{R}^{3 \times 3}$:

$$h(\mathbf{x}) = \frac{e^{-\frac{1}{2}\mathbf{x}^T\Sigma^{-1}\mathbf{x}}}{\sqrt{(2\pi)^3 \det \Sigma}}, \quad \mathbf{x} \in \mathbb{R}^3. \quad (2)$$

The FT $\hat{h}$ of $h$ is given by: $\hat{h}(\xi) = e^{-\frac{1}{2}\xi^T\Sigma^{-1}\xi}$, where $\xi \in \mathbb{R}^3$ is the wavevector. In this framework, inferring the PSF consists of optimizing the coefficients of the symmetric positive semi-definite covariance matrix $\Sigma$, which directly controls the level of blur in the rendered image. More photorealistic PSFs based on differentiable physical models could also be incorporated [29, 30], allowing the learned parameters to correspond to real optical properties, such as refractive indices or optical aberrations expressed using Zernike polynomials [47].

## 3.3. Geometric models of biological objects

Extracting geometry from volumetric images involves approximating biological structures using discrete N-dimensional representations ($N = 0, 1, 2, 3$) embedded in

3D space. These structures vary in size and topology. We focus here on cell membranes, modeled as 2D surfaces using triangle meshes, though extensions to points, filaments, and volumes are possible. This choice reflects prior knowledge of structural topology: as shown in Section 6.4, early embryos comprise cells forming bounded regions, representable as a single non-manifold, multitematerial mesh [15, 56], where cell-cell interfaces are captured by doubling intensity at overlapping membranes.

# 4. Fourier transform of triangulated surfaces

We derive an explicit expression for the FT of arbitrary 2D surfaces embedded in $\mathbb{R}^3$, and provide closed-form formulas for both the FT and its gradient with respect to vertex positions in the case of a triangulated mesh.

## 4.1. Surfaces as spatial Dirac distributions

We model the spatial fluorophore density $u_Λ$ as a Dirac distribution supported on the surface $\Lambda$, weighted by the local surface element to reflect the actual fluorophore density along the surface.

<figure><img src="image_4.png" /><figcaption>Figure 4. Surface $\Lambda$ describing a dividing cell and its associated fluorophore density $u_Λ$ convolved with a Gaussian PSF in $\mathbb{R}^3$ (maximum projection).</figcaption></figure>

We consider a surface parametrized by a function $\Lambda: [0, 1]^2 \to [0, 1]^3$, mapping local coordinates $(s_1, s_2)$, defined in a tangent orthonormal basis, to points in 3D space. The spatial fluorophore density in $\mathbb{R}^3$ is then defined as:

$$u_{\Lambda}(\mathbf{x}) = \frac{1}{|\Lambda|} \iint \delta_{\Lambda}(s_1, s_2)(\mathbf{x}) a_{\Lambda}(s_1, s_2) \, ds_1 ds_2, \quad (3)$$

where $\delta_{\Lambda}(s_1, s_2)$ denotes the Dirac delta function centered at $\Lambda(s_1, s_2)$, $a_Λ = \|\partial_{s_1}\Lambda \times \partial_{s_2}\Lambda\|$ is the surface element. The normalization by the total surface area $|\Lambda| = \iint a_Λ(s_1, s_2) ds_1 ds_2$ ensures that the total integrated image intensity is one. When dealing with several surfaces with different densities, one can use a weighted sum or integral of $u_Λ$. Cell-cell interfaces are modeled as double membranes, and their fluorophore density is therefore scaled by a factor two.

For all $\mathbf{x} = (x, y, z) \in \mathbb{R}^3$, the FT of the Dirac distribution at a point $\mathbf{x}$ is given by $\hat{\delta}_{\mathbf{x}}(\xi) = e^{-i\mathbf{x}\cdot\xi}$, $\xi \in \mathbb{R}^3$. By linearity, the FT of $u_Λ$ reads therefore:

$$\hat{u}_{\Lambda}(\xi) = \frac{1}{|\Lambda|} \iint a_{\Lambda}(s_1, s_2) e^{-i\Lambda(s_1, s_2)\cdot\xi} \, ds_1 ds_2. \quad (4)$$
In practice, we discretize $\Lambda$ with triangle meshes. By linearity, this reduces to computing and summing the FT of each triangle.

### 4.1.1. Spatial density for a triangle mesh

A triangle mesh is a surface $\Lambda = \{\mathcal{T}\}$ defined by a set of triangles. For $\xi \in \mathbb{R}^3$, by linearity, its FT is defined by:

$$ \hat{u}_{\Lambda}(\xi) = \frac{\sum_{\mathcal{T} \in \Lambda} \hat{u}_{\mathcal{T}}(\xi)}{|\Lambda|} = \frac{\sum_{\mathcal{T} \in \Lambda} \hat{u}_{\mathcal{T}}(\xi)}{\sum_{\mathcal{T} \in \Lambda} A_{\mathcal{T}}}, \quad (5) $$

Where $A_{\mathcal{T}}$ is the area of $\mathcal{T}$. The gradient of the FT with respect to a vertex $\mathbf{v}$ reads:

$$ \frac{\partial \hat{u}_{\Lambda}(\xi)}{\partial \mathbf{v}} = \frac{1}{|\Lambda|} \left( \sum_{\mathcal{T} \in \mathbf{v}^*} \frac{\partial \hat{u}_{\mathcal{T}}(\xi)}{\partial \mathbf{v}} - \hat{u}_{\Lambda}(\xi) \sum_{\mathcal{T} \in \mathbf{v}^*} \frac{\partial A_{\mathcal{T}}}{\partial \mathbf{v}} \right), \quad (6) $$

where $\mathbf{v}^*$ denotes the set of the triangles of $\Lambda$ that contains the vertex $\mathbf{v}$.

<figure><img src="image_5.png" /><figcaption>Figure 5. Notations for a triangle mesh.</figcaption></figure>

### 4.1.2. Fourier transform and gradients of a triangle

We consider a triangle $\mathcal{T}$ defined by its vertices $(\mathbf{v}_1, \mathbf{v}_2, \mathbf{v}_3)$ and compute $\hat{u}_{\mathcal{T}}$ and its spatial derivatives. For convenience, we set $\mathbf{v}_4 = \mathbf{v}_1$ and similarly by permutation $\mathbf{v}_0 = \mathbf{v}_3$. For $p=1...3$, we define $p^- = p-1$ and $p^+ = p+1$ and denote by $\mathbf{e}_p = \mathbf{v}_{p-} - \mathbf{v}_{p+}$ the opposite edge to $\mathbf{v}_p$ and by $l_p = |\mathbf{e}_p|$ its length. $A_{\mathcal{T}} = \frac{|\mathbf{e}_3 \times \mathbf{e}_1|}{2}$ denotes the area of the triangle, and $\mathbf{N}_{\mathcal{T}} = \frac{\mathbf{e}_3 \times \mathbf{e}_1}{2A_{\mathcal{T}}}$ denotes its unit normal. For $p=1...3$, we define $\mathbf{w}_p = \mathbf{e}_p \times \mathbf{N}_{\mathcal{T}}$, the non-normalized outward normal to $\mathcal{T}$ on the edge $\mathbf{e}_p$. This allows us expressing the gradient of the triangle area with respect to each vertex position as $\frac{\partial A_{\mathcal{T}}}{\partial \mathbf{v}_p} = -\frac{\mathbf{w}_p}{2}$, for $p=1...3$.

Expressing the FT of the density on a triangle as $\hat{u}_{\mathcal{T}}(\xi) = \int_{\mathcal{T}} e^{-i\xi \cdot \mathbf{x}} ds(\mathbf{x})$, then for all $\xi \in \mathbb{R}^3$, we have:

$$ \hat{u}_{\mathcal{T}}(\xi) = 2A_{\mathcal{T}} f_{\mathcal{T}}(\xi), \quad (7) $$

$$ \text{with } f_{\mathcal{T}}(\xi) = \sum_{p=1}^{3} \frac{e^{-i \mathbf{v}_p \cdot \xi}}{(\mathbf{e}_{p-} \cdot \xi)(\mathbf{e}_{p+} \cdot \xi)}. \quad (8) $$

We deduce

$$ \frac{\partial \hat{u}_{\mathcal{T}}(\xi)}{\partial \mathbf{v}_p} = -f_{\mathcal{T}}(\xi) \mathbf{w}_p + 2A_{\mathcal{T}} \frac{\partial f_{\mathcal{T}}(\xi)}{\partial \mathbf{v}_p}, \quad p = 1...3 \quad (9) $$

$$ \text{with } \frac{\partial f_{\mathcal{T}}(\xi)}{\partial \mathbf{v}_p} = \xi \left[ \frac{e^{-i \mathbf{v}_{p+} \cdot \xi}}{(\mathbf{e}_{p-} \cdot \xi)^2 (\mathbf{e}_p \cdot \xi)} - \frac{e^{-i \mathbf{v}_{p-} \cdot \xi}}{(\mathbf{e}_p \cdot \xi)(\mathbf{e}_{p+} \cdot \xi)^2} - \frac{\text{ie}^{-i \mathbf{v}_{p} \cdot \xi}}{(\mathbf{e}_{p-} \cdot \xi)(\mathbf{e}_{p+} \cdot \xi)} + \frac{e^{-i \mathbf{v}_{p} \cdot \xi}}{(\mathbf{e}_{p-} \cdot \xi)^2 (\mathbf{e}_{p+} \cdot \xi)} - \frac{e^{-i \mathbf{v}_{p} \cdot \xi}}{(\mathbf{e}_{p-} \cdot \xi)(\mathbf{e}_{p+} \cdot \xi)^2} \right]. \quad (10) $$

### 4.1.3. Numerical approximations to prevent divergence

The Fourier Transform (FT) of a Dirac distribution on a triangle is $C^{\infty}$. However, significant computational errors arise when a denominator in Eq. (7) approaches zero. Due to rounding errors in floating-point arithmetic, numerical precision is limited to a threshold $\epsilon$, below which values cannot be reliably distinguished from zero. To mitigate this issue, when a denominator term in Eq. (7) approaches zero, we substitute it with a stable approximation, which we detail in the following section. We write $f_{\mathcal{T}}(\xi) = g(\mathbf{e}_1 \cdot \xi, \mathbf{e}_2 \cdot \xi, \mathbf{e}_3 \cdot \xi)$, with the function $g$ defined for $(s, t, u) \in \mathbb{R}^3$ by

$$ g(s, t, u) = \frac{-e^{is}}{(s-t)(s-u)} + \frac{-e^{it}}{(t-u)(t-s)} + \frac{-e^{iu}}{(u-s)(u-t)}. \quad (11) $$

When two values $(a, b)$ among $(s, t, u)$ satisfy $|a-b| < \epsilon$, the denominator in Eq. (11) approaches zero, leading to divergence. To prevent this, we derive exact expressions for $g(s, t, u)$ in the special cases where $t=u$, $t=s$, $u=s$, or $u=s=t$. These alternative expressions replace the original formulation in Eq. (11) whenever $|a-b| < \epsilon$, ensuring numerical stability:

$$ \begin{aligned} g(t, t, u) &= g(u, t, t) = g(t, u, t) \\ &= i \frac{e^{-it}}{t-u} + \frac{e^{-it}}{(t-u)^2} - \frac{e^{-iu}}{(t-u)^2}. \end{aligned} \quad (12a) $$

$$ g(u, u, u) = \frac{e^{-iu}}{2}. \quad (12b) $$

# 5. Numerical implementation

## 5.1. Acceleration

Computing the artificial image requires evaluating the FT $\hat{u}_{\Lambda}$ for each of the $N$ voxels times the number $n_t$ of triangles in the mesh, resulting in a runtime complexity of $O(N \cdot n_t)$. The gradient of the mesh FT involves an $O(1)$ sum per voxel for each of the $n_v$ vertices, leading to a total complexity of $O(N \cdot n_v)$. In practice, a confocal microscopy
image of size 500³ contains 125 million voxels, while a reasonable mesh typically consists of $10^3-4$ vertices and triangles. Without proper parallelization, these computations become prohibitive. The high computational cost has been a key limitation preventing the widespread use of spectral methods in 3D, despite their promising applications [37].

We propose two complementary strategies to accelerate the mesh FT: GPU parallelization and a narrow-band approximation method in the frequency domain.

**GPU parallelization** The grid-based structure of a 3D image naturally lends itself to massively parallel computations on graphical processing units (GPUs). To leverage this, we provide a custom CUDA implementation for both, the forward and the backward passes. As shown in Figure 6, our implementation achieves ≈ $10^3$ speedup on one NVIDIA V100 GPU compared to a vectorized CPU implementation, while also enabling computations on significantly larger box sizes before encountering overflow.

<figure><img src="image_6.png" /><figcaption>Figure 6. Benchmarking of Mesh FT Computation: (Left) For a 80-triangle spherical mesh, our CUDA implementation significantly improves speed and memory efficiency as function of image size. (Right) For an image of size [100]³, the forward pass scales linearly with the number of triangles.</figcaption></figure>

## Narrow-band approximation in the frequency domain

As previously stated, the FT of the image is given by the element-wise product of the FT of the mesh and the PSF: $\hat{I} = \hat{u}_{\Lambda} \cdot \hat{h}$. In practice, the PSF acts as a low-pass filter, emphasizing low spatial frequencies of the mesh. A blurred image corresponds to a sparse PSF, with its highest amplitudes concentrated near zero frequency, thereby suppressing fine mesh details. The more resolved the original biological image, the more frequency components of the PSF are required for accurate rendering. For any given spatial frequency $\xi$, if $\hat{h}(\xi) \approx 0$, then computing $\hat{u}_{\Lambda}(\xi)$ is unnecessary, as it contributes negligibly to $\hat{I}(\xi)$ and hence $I(x)$. To exploit this, we apply a frequency-domain cutoff, computing the FT of the mesh only for high-frequency, low-amplitude regions of the PSF (only where its FT exceeds 1% of its maximum value). This spectral narrow-band method reduces computational cost by several orders of magnitude, skipping computations with negligible far-field contributions in real space.

**A staggered optimization scheme** Despite significant speed improvements, computing the mesh FT and its gradient remains the main computational bottleneck of our pipeline. We offer two approaches for computing the FT: a fast, approximate method using the narrow-band approach and a slower, exact method performing the full computation. The forward pass is approximately 10 times slower than the backward pass, making it crucial to apply the narrow-band method in the backward computation whenever possible. To address this, we decompose the coupled optimization problem into two staggered sub-problems, iterating them sequentially. (a) Shape optimization: To optimize the shape, an approximate computation of the mesh FT suffices. We use the narrow-band method to efficiently compute the FT and perform an optimization step for the vertex positions while keeping the PSF fixed. (b) PSF optimization: Optimizing the PSF requires computing the mesh FT across all available frequencies, avoiding bias from the narrow-band threshold, which depends on the current PSF values. Thus, we compute the exact FT of the mesh and perform an optimization step for the PSF parameters while keeping the shape fixed.

We optimize shapes by minimizing a weighted $L^2$ loss $\mathcal{L} = \langle(I - I_{GT})^2, I_{GT}\rangle$, where $\langle \cdot, \cdot \rangle$ denotes the Frobenius inner product. This formulation downweights dark regions in the ground-truth image $I_{GT}$ to focus the optimization on informative signal. Following [65], we update mesh vertex positions using the AdamUniform optimizer and apply a diffusion-based regularization to the gradient: $v \leftarrow v - \eta (1 + \lambda L)^{-2} \frac{\partial \mathcal{L}}{\partial v}$, where $L \in \mathbb{R}^{n_v \times n_v}$ is the mesh Laplacian and $\lambda = 50$. Computing $(1 + \lambda L)^{-2}$ explicitly is memory-intensive for large vertex counts $n_v$, so we instead solve the equivalent linear system $(1 + \lambda L)^{-2} A = \frac{\partial \mathcal{L}}{\partial v}$ via GPU-parallelized sparse Cholesky decomposition [63, 65].

## 5.2. PSF optimization step

PSF parameter optimization aims to accurately reconstruct both bright and dark regions of the image. A regular $L^2$ norm minimization, $\mathcal{L}_0 = \|I - I_{GT}\|^2$ is well-suited for this task. In this step, we optimize the covariance matrix $\Sigma$ using the Adam optimizer [43].

# 6. Benchmarking and application

## 6.1. Benchmarks on artificial images

We first benchmark our method on artificial images generated from common computer graphics meshes and a simulated dividing cell (Figure 7). Images are rendered using a Gaussian isotropic PSF, which was also used for inference to isolate the effect of shape reconstruction. Starting from elementary mesh shapes (a sphere or torus, depending on topology), we perform 10,000 optimization iterations per example. Despite the absence of remeshing or collision de-
tection algorithms [10, 52], our regularized shape optimization accurately converges to the original shapes.

<figure><img src="image_7.png" /><figcaption>Figure 7. Shape inference from 3D artificial images: Starting from spherical or toroidal meshes, the optimization process reconstructs meshes that closely align with the original shapes (see also: Supplemental videos 1–4).</figcaption></figure>

## 6.2. Robustness to initial conditions and noise

Next, we evaluate the convergence robustness to initial conditions and noise. As a metric, we compute the mean-squared error (MSE) between the reconstructed shape—rendered as an artificial image—and the target image generated from the initial mesh. Using Spot (cow) as the reference shape, we assess the MSE when the starting mesh is scaled (zoomed in/out) and shifted from its barycenter (Figure 8, top and middle). While our method is generally robust to initialization, large shifts from the target increase the number of iterations needed for convergence. Notably, our approach demonstrates high robustness to random noise added to the image (Figure 8, bottom), which is particularly advantageous for microscopy applications.

## 6.3. Benchmarking against an active mesh method

Traditional active mesh methods were primarily developed for segmenting bright volumetric regions [12, 21], making them less suited for structures characterized by narrow intensity bands. We benchmark our approach against DM3D (Deforming Mesh 3D) [75], a recent active mesh algorithm designed to detect cell interfaces in fluorescence microscopy images. Using the authors' dataset of 15 mouse organoid 3D images with corresponding inferred

<figure><img src="image_8.png" /><figcaption>Figure 8. Benchmarking robustness to initialization and noise. (Top, Middle) MSE evaluation for shape reconstruction using Spot as the reference mesh, with variations in initial mesh scale and position. Larger shifts increase convergence time. (Bottom) The method is quite robust to random noise.</figcaption></figure>

<figure><img src="image_9.png" /><figcaption>Figure 9. Benchmarking *deltaMic* vs DM3D [75] on 3D mouse organoid images.</figcaption></figure>

meshes, we reapply *deltaMic* to the original image volumes. We then compare the outputs of DM3D and *deltaMic* using mean squared error (MSE), structural similarity index (SSIM) [82], and peak signal-to-noise ratio (PSNR) [33],
<figure><img src="image_10.png" /><figcaption>Figure 10. Cell shape inference in early embryos: Starting from a foam-like mesh, both vertex positions and PSF are optimized to infer individual cell shapes in 4-cell mouse (top), 8-cell ascidian (middle), and 16-cell worm (bottom) embryos, using 3D fluorescence images acquired respectively with single-view, dual-view light-sheet, and point-scanning microscopes (see also: Supplemental videos 5–9). Mouse and ascidian data - courtesy of K. Cavanaugh and D. Gonzalez-Suarez.</figcaption></figure>

computed between the reconstructed shapes rendered as synthetic images with our inferred PSF. Our method consistently outperforms DM3D across all metrics (Figure 9).

## 6.4. Shape inference from embryo microscopy

We applied our method to infer cell shapes in early-stage embryos from different species and 3D fluorescence microscopy modalities: mouse (single-view light-sheet), ascidian (dual-view light-sheet), and worm (point-scanning confocal) [11]. Cell clusters were represented as non-manifold multimaterial meshes [9, 25, 56], and we used a multimaterial mesh-based surface tracking method [15] to handle remeshing, collision detection, and topological transitions. As an initial guess, we generated a foam-like multimaterial mesh with the correct number of cells. The resulting cell clusters, ranging between 1000–10000 vertices, were optimized to fit the microscopy data. We jointly optimized the PSF and mesh following the strategy in Section 5.1. *deltaMic* accurately reconstructed cell shapes in 4-, 8-, and 16-cell embryos, converging within 1k and 10k optimization steps, yielding rendered images that closely matched the original microscopy data (Figure 10).

## 7. Conclusion

We introduced *deltaMic*, a first-of-its-kind differentiable 3D microscopy image creator that renders realistic 3D confocal fluorescence images from a surface mesh and a parametrized PSF. Our GPU-parallelized mesh FT implementation enables efficient forward and backward computations, further optimized with a spectral narrow-band method. *deltaMic* is relatively robust to random noise in the

image and mesh initialization, accurately recovering fine details of complex shapes from artificial and real 3D microscopy images thanks to its ability to fine-tune the shape loss function via the PSF. Unlike traditional active mesh methods, it does not require fine-tuned shape regularization terms to achieve smooth results. However, knowledge-based priors can be incorporated to constrain cell shapes, opening the avenue for direct implementation of inverse mechanical problems from microscopy data [34] through integration with a differentiable physical simulator.

Our approach opens numerous avenues for both practical applications and theoretical extensions. As an active contour method, it naturally lends itself to biological shape tracking in time-lapse microscopy. Incorporating more photorealistic PSF models tailored to specific microscopy modalities [1, 29, 30, 47] could enable blind deconvolution of microscopy images with integrated shape priors [17, 49]. But extending our framework to handle space-variant PSFs [16, 22], which violate translational invariance, may require significant theoretical and computational developments. In terms of geometry, the method could be adapted to handle 1D structures—such as cytoskeletal filaments, whose segmentation remains a major challe [67]—or extended to fully 3D shapes like nuclei or condensates, paving the way for multidimensional active mesh algorithms. On the implementation side, additional acceleration strategies could help mitigate the computational demands of large 3D volumes. Finally, thanks to its full differentiability, our method can be readily integrated as a module within larger (deep-)learning pipelines [39, 60].
## 8. Acknowledgments

SI, AS, FB and HT are supported by the CNRS and Collège de France. HT received funding from the European Research Council (ERC) under the European Union's Horizon 2020 research and innovation programme (Grant agreement No. 949267) and from the French Agence Nationale de la Recherche (Grants ANR-22-CE13-0036 and ANR-23-CE30-0013). SI fellowship was funded by Ecole Polytechnique (AMX grant). We thank Baptiste Nicolet for insightful discussions, H. Borja da Rocha and K. Crane for sharing meshes, K. Cavanaugh and D. Gonzalez Suarez for sharing fluorescence microscopy images of mouse and ascidian embryos.

## References

[1] François Aguet, Stefan Geissbühler, Iwan Märki, Theo Lasser, and Michael Unser. Super-resolution orientation estimation and localization of fluorescent dipoles using 3-d steerable filters. *Optics express*, 17(8):6829–6848, 2009. 2, 8

[2] Matthew R. Arnison and Colin J.R. Sheppard. A 3d vectorial optical transfer function suitable for arbitrary pupil functions. *Optics Communications*, 211(1):53–63, 2002. 2

[3] Chayan Banerjee, Kien Nguyen, Clinton Fookes, and Karaniadakis George. Physics-informed computer vision: A review and perspectives. *ACM Computing Surveys*, 57(1):1–38, 2024. 1

[4] Joshua Batson and Loïc Royer. Noise2self: Blind denoising by self-supervision. In *Proceedings of the 36th International Conference on Machine Learning*, ICML 2019, 9-15 June 2019, Long Beach, California, USA, pages 524–533. PMLR, 2019. 1

[5] Chinmay Belthangady and Loic A Royer. Applications, promises, and pitfalls of deep learning for fluorescence image reconstruction. *Nature methods*, 16(12):1215–1225, 2019. 1

[6] Serge Beucher. The watershed transformation applied to image segmentation. *Scanning Microscopy*, 1992(6):28, 1992. 3

[7] Serge Beucher and Fernand Meyer. The morphological approach to segmentation: the watershed transformation. In *Mathematical morphology in image processing*, pages 433–481. CRC Press, 2018. 3

[8] Yuri Boykov and Gareth Funka-Lea. Graph cuts and efficient nd image segmentation. *International journal of computer vision*, 70(2), 2006. 3

[9] Kenneth A Brakke. The surface evolver. *Experimental mathematics*, 1(2):141–165, 1992. 8

[10] Tyson Brochu, Essex Edwards, and Robert Bridson. Efficient geometrically exact continuous collision detection. *ACM Transactions on Graphics (TOG)*, 31(4):1–7, 2012. 7

[11] Jianfeng Cao, Guoye Guan, Vincy Wing Sze Ho, Ming-Kin Wong, Lu-Yan Chan, Chao Tang, Zhongying Zhao, and Hong Yan. Establishment of a morphological atlas of the caenorhabditis elegans embryo using deep-learning-based 4d segmentation. *Nature communications*, 11(1):1–14, 2020. 8

[12] T.F. Chan and L.A. Vese. Active contours without edges. *IEEE Transactions on Image Processing*, 10(2):266–277, 2001. 1, 3, 7

[13] Özgün Çiçek, Ahmed Abdulkadir, Soeren S. Lienkamp, Thomas Brox, and Olaf Ronneberger. 3d u-net: Learning dense volumetric segmentation from sparse annotation. In *Medical Image Computing and Computer-Assisted Intervention – MICCAI 2016*, pages 424–432, Cham, 2016. Springer International Publishing. 1, 3

[14] Kevin J Cutler, Carsen Stringer, Teresa W Lo, Luca Rappez, Nicholas Stroustrup, S Brook Peterson, Paul A Wiggins, and Joseph D Mougous. Omnipose: a high-precision morphology-independent solution for bacterial cell segmentation. *Nature methods*, 19(11):1438–1448, 2022. 3

[15] Batty C. & Grinspun E. Da, F. Multimaterial mesh-based surface tracking. *ACM Trans. Graph.*, 33:112:1–112:11, 2014. 4, 8

[16] Valentin Debarnot and Pierre Weiss. Deepblur: Blind identification of space variant psf. In *2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI)*, pages 1544–1547. IEEE, 2021. 8

[17] Valentin Debarnot and Pierre Weiss. Deep-blur: Blind identification and deblurring with convolutional neural networks. *Biological Imaging*, 4:e13, 2024. 8

[18] Serge Dmitrieff and François Nédélec. Confocalgn: A minimalistic confocal image generator. *SoftwareX*, 6:243–247, 2017. 2

[19] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. *arXiv preprint arXiv:2010.11929*, 2020. 1

[20] Alexandre Dufour, Vasily Shinin, Shahragim Tajbakhsh, Nancy Guillén-Aghion, J-C Olivo-Marin, and Christophe Zimmer. Segmenting and tracking fluorescent cells in dynamic 3-d microscopy with coupled active surfaces. *IEEE Transactions on Image Processing*, 14(9):1396–1410, 2005. 1

[21] Alexandre Dufour, Roman Thibeaux, Elisabeth Labruyere, Nancy Guillen, and Jean-Christophe Olivo-Marin. 3-d active meshes: fast discrete deformable models for cell tracking in 3-d time-lapse microscopy. *IEEE transactions on image processing*, 20(7):1925–1937, 2010. 1, 3, 7

[22] Paul Escande and Pierre Weiss. Sparse wavelet representations of spatially varying blurring operators. *SIAM Journal on Imaging Sciences*, 8(4):2976–3014, 2015. 8

[23] Dennis Eschweiler, Malte Rethwisch, Mareike Jarchow, Simon Koppers, and Johannes Stegmaier. 3d fluorescence microscopy data synthesis for segmentation and benchmarking. *Plos one*, 16(12):e0260509, 2021. 2

[24] Romain Fernandez, Pradeep Das, Vincent Mirabet, Eric Moscardi, Jan Traas, Jean-Luc Verdeil, Grégoire Malandain, and Christophe Godin. Imaging plant growth in 4d: robust tissue reconstruction and lineaging at cell resolution. *Nature methods*, 7(7):547–553, 2010. 3
[25] Julie Firmin, Nicolas Ecker, Diane Rivet Danon, Özge Özgüç, Virginie Barraud Lange, Hervé Turlier, Catherine Patrat, and Jean-Léon Maître. Mechanics of human embryo compaction. *Nature*, 629(8012):646–651, 2024. 8

[26] Roy Frostig, Matthew Johnson, and Chris Leary. Compiling machine learning programs via high-level tracing. 2018. 2, 3

[27] Sarah Frisken Gibson and Frederick Lanni. Experimental test of an analytical model of aberration in an oil-immersion objective lens used in three-dimensional light microscopy. *J. Opt. Soc. Am. A*, 9(1):154–166, 1992. 2

[28] Miller-G. Moen E. et al. Greenwald, N.F. Whole-cell segmentation of tissue images with human-level performance using large-scale data annotation and deep learning. *Nat Biotechnol*, 40:555–565, 2022. 3

[29] Bridget M. Hanser, Mats G. L. Gustafsson, David A. Agard, and John W. Sedat. Phase retrieval for high-numerical-aperture optical systems. *Opt. Lett.*, 28(10):801–803, 2003. 2, 4, 8

[30] B. M. Hanser, M. G. L. Gustafsson, D. A. Agard, and J. W. Sedat. Phase-retrieved pupil functions in wide-field fluorescence microscopy. *Journal of Microscopy*, 216(1):32–48, 2004. 2, 4, 8

[31] Jo A. Helmuth and Ivo F. Sbalzarini. Deconvolving active contours for fluorescence microscopy images. In *Advances in Visual Computing*, pages 544–553, Berlin, Heidelberg, 2009. Springer Berlin Heidelberg. 3

[32] Reka Hollandi, Abel Szklalisity, Timea Toth, Ervin Tasnadi, Csaba Molnar, Botond Mathe, Istvan Grexa, Jozsef Molnar, Arpad Balind, Mate Gorbe, Maria Kovacs, Ede Migh, Allen Goodman, Tamas Balassa, Krisztian Koos, Wenyu Wang, Juan Carlos Caicedo, Norbert Bara, Ferenc Kovacs, Lassi Paavolainen, Tivadar Danka, Andras Kriston, Anne Elizabeth Carpenter, Kevin Smith, and Peter Horvath. nucleaizer: A parameter-free deep learning framework for nucleus segmentation using image style transfer. *Cell Systems*, 10(5): 453–458.e6, 2020. 2

[33] Quang Huynh-Thu and Mohammed Ghanbari. Scope of validity of psnr in image/video quality assessment. *Electronics Letters*, 44(13):800–801, 2008. 7

[34] Sacha Ichbiah, Fabrice Delbary, Alex McDougall, Rémi Dumollard, and Hervé Turlier. Embryo mechanics cartography: inference of 3d force atlases from fluorescence microscopy. *Nature methods*, 20(12):1989–1999, 2023. 1, 8

[35] Sacha Ichbiah, Fabrice Delbary, Anshuman Sinha, and Hervé Turlier. deltamic: differentiable microscopy image creator for cell shape inference with active mesh (1.0.0). *Zenodo*. https://doi.org/10.5281/zenodo.16623962, 2025. 2

[36] Shinya Inoué. *Foundations of Confocal Scanned Imaging in Light Microscopy*, pages 1–19. Springer US, Boston, MA, 2006. 2

[37] Chiyu Jiang, Dequan Wang, Jingwei Huang, Philip Marcus, Matthias Nießner, et al. Convolutional neural networks on non-uniform geometrical signals using euclidean spectral transformation. *arXiv preprint arXiv:1901.02070*, 2019. 6

[38] Michael Kass, Andrew P. Witkin, and Demetri Terzopoulos. Snakes: Active contour models. *Int. J. Comput. Vis.*, 1(4): 321–331, 1988. 1, 3

[39] Hiroharu Kato, Yoshitaka Ushiku, and Tatsuya Harada. Neural 3d mesh renderer. In *2018 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2018*, Salt Lake City, UT, USA, June 18-22, 2018, pages 3907–3916. Computer Vision Foundation / IEEE Computer Society, 2018. 3, 8

[40] Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. *CoRR*, abs/2006.12057, 2020. 3

[41] Hiroharu Kato, Deniz Beker, Mihai Morariu, Takahiro Ando, Toru Matsuoka, Wadim Kehl, and Adrien Gaidon. Differentiable rendering: A survey. *arXiv preprint arXiv:2006.12057*, 2020. 2

[42] Philipp J Keller. Imaging morphogenesis: technological advances and biological insights. *Science*, 340(6137):1234168, 2013. 2

[43] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. *arXiv preprint arXiv:1412.6980*, 2014. 6

[44] Cheveralls K.C. Leonetti M.D. et al. Kobayashi, H. Self-supervised deep learning encodes high-resolution features of protein subcellular localization. *Nat Methods*, 19:995–1003, 2022. 1

[45] Hal G. Kraus. Huygens–fresnel–kirchhoff wave-front diffraction formulation: spherical waves. *J. Opt. Soc. Am. A*, 6(8):1196–1205, 1989. 2

[46] Samuli Laine, Janne Hellsten, Tero Karras, Yeongho Seol, Jaakko Lehtinen, and Timo Aila. Modular primitives for high-performance differentiable rendering. *ACM Trans. Graph.*, 39(6), 2020. 3

[47] Vasudevan Lakshminarayanan and Andre Fleck. Zernike polynomials: a guide. *Journal of Modern Optics*, 58(7):545–561, 2011. 4, 8

[48] Jaakko Lehtinen, Jacob Munkberg, Jon Hasselgren, Samuli Laine, Tero Karras, Miika Aittala, and Timo Aila. Noise2noise: Learning image restoration without clean data. In *Proceedings of the 35th International Conference on Machine Learning*, ICML 2018, *Stockholmsmässan*, Stockholm, Sweden, July 10-15, 2018, pages 2971–2980. PMLR, 2018. 1

[49] Anat Levin, Yair Weiss, Fredo Durand, and William T Freeman. Understanding and evaluating blind deconvolution algorithms. In *2009 IEEE conference on computer vision and pattern recognition*, pages 1964–1971. IEEE, 2009. 8

[50] Tzu-Mao Li, Michal Lukáč, Michaël Gharbi, and Jonathan Ragan-Kelley. Differentiable vector graphics rasterization for editing and learning. *ACM Trans. Graph.*, 39(6), 2020. 3

[51] Jeff W Lichtman and José-Angel Conchello. Fluorescence microscopy. *Nature methods*, 2(12):910–919, 2005. 1

[52] Ming C Lin, Dinesh Manocha, and Young J Kim. Collision and proximity queries. In *Handbook of discrete and computational geometry*, pages 1029–1056. Chapman and Hall/CRC, 2017. 7

[53] Shichen Liu, Tianye Li, Weikai Chen, and Hao Li. Soft rasterizer: A differentiable renderer for image-based 3d reasoning. In *Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)*, 2019. 3
[54] Jun Ma, Yuting He, Feifei Li, Lin Han, Chenyu You, and Bo Wang. Segment anything in medical images. *Nature Communications*, 15(1):654, 2024. 1, 3

[55] Mercier V. Chiaruttini N. Machado, S. Limeseg: a coarse-grained lipid membrane simulation for 3d image segmentation. *BMC Bioinformatics*, 20, 2019. 1, 3

[56] Jean-Léon Maître, Hervé Turlier, Rukshala Illukkumbura, Björn Eismann, Ritsuya Niwayama, François Nédélec, and Takashi Hiiragi. Asymmetric division of contractile domains couples cell positioning and fate specification. *Nature*, 536 (7616):344–348, 2016. 4, 8

[57] Ravi Malladi, James A Sethian, and Baba C Vemuri. Shape modeling with front propagation: A level set approach. *IEEE transactions on pattern analysis and machine intelligence*, 17(2):158–175, 1995. 3

[58] Patrik Malm, Anders Brun, and Ewert Bengtsson. Simulation of bright-field microscopy images depicting pap-smear specimen. *Cytometry Part A*, 87(3):212–226, 2015. 2

[59] Erik Meijering. Cell segmentation: 50 years down the road [life sciences]. *IEEE Signal Processing Magazine*, 29(5): 140–145, 2012. 3

[60] Ben Mildenhall, Pratul P. Srinivasan, Matthew Tancik, Jonathan T. Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. *CoRR*, abs/2003.08934, 2020. 8

[61] Leonid Mill, David Wolff, Nele Gerrits, Patrick Philipp, Lasse Kling, Florian Vollnhals, Andrew Ignatenko, Christian Jaremenko, Yixing Huang, Olivier De Castro, et al. Synthetic image rendering solves annotation problem in deep learning nanoparticle segmentation. *Small Methods*, 5(7):2100223, 2021. 2

[62] David Bryant Mumford and Jayant Shah. Optimal approximations by piecewise smooth functions and associated variational problems. *Communications on pure and applied mathematics*, 1989. 3

[63] Maxim Naumov. Parallel solution of sparse triangular linear systems in the preconditioned iterative methods on the gpu. *NVIDIA Corp.*, Westford, MA, USA, Tech. Rep. NVR-2011, 1, 2011. 6

[64] Jeffrey P. Nguyen, Benjamin P. Bratton, and Joshua W. Shae-vitz. *Biophysical Measurements of Bacterial Cell Shape*, pages 227–245. Springer New York, New York, NY, 2016. 3

[65] Baptiste Nicolet, Alec Jacobson, and Wenzel Jakob. Large steps in inverse rendering of geometry. *ACM Trans. Graph.*, 40(6), 2021. 2, 6

[66] Merlin Nimier-David, Delio Vicini, Tizian Zeltner, and Wenzel Jakob. Mitsuba 2: A retargetable forward and inverse renderer. *ACM Trans. Graph.*, 38(6), 2019. 3

[67] Bugra Özdemir and Ralf Reski. Automated and semi-automated enhancement, segmentation and tracing of cytoskeletal networks in microscopic images: A review. *Computational and Structural Biotechnology Journal*, 19:2106–2120, 2021. 8

[68] Stringer C. Pachitariu, M. Cellpose 2.0: how to train your own model. *Nat Methods*, 19:1634–1641, 2022. 3

[69] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming

Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. *PyTorch: An Imperative Style, High-Performance Deep Learning Library*. Curran Associates Inc., Red Hook, NY, USA, 2019. 2, 3

[70] Satwik Rajaram, Benjamin Pavie, Nicholas EF Hac, Steven J Altschuler, and Lani F Wu. Simucell: a flexible framework for creating synthetic microscopy images. *Nature methods*, 9(7):634–635, 2012. 2

[71] Nikhila Ravi, Jeremy Reizenstein, David Novotný, Taylor Gordon, Wan-Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. *CoRR*, abs/2007.08501, 2020. 3

[72] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. Unet: Convolutional networks for biomedical image segmentation. In *Medical Image Computing and Computer-Assisted Intervention – MICCAI 2015*, pages 234–241, Cham, 2015. Springer International Publishing. 1, 3

[73] Daniel Sage, Lauréne Donati, Ferréol Soulez, Denis Fortun, Guillaume Schmit, Arne Seitz, Romain Guiet, Cédric Vonesch, and Michael Unser. Deconvolutionlab2: An open-source software for deconvolution microscopy. *Methods*, 115:28–41, 2017. 2

[74] Uwe Schmidt, Martin Weigert, Coleman Broaddus, and Gene Myers. Cell detection with star-convex polygons. In *Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 - 21st International Conference, Granada, Spain, September 16-20, 2018, Proceedings, Part II*, pages 265–273, 2018. 1

[75] Matthew B Smith, Hugh Sparks, Jorge Almagro, Agathe Chaigne, Axel Behrens, Chris Dunsby, and Guillaume Salbreux. Active mesh and neural network pipeline for cell aggregate segmentation. *Biophysical journal*, 122(9):1586–1599, 2023. 1, 3, 7

[76] Ahmet Can Solak, Loic A. Royer, Abdur-Rahmaan Janhangeer, and Hirofumi Kobayashi. royerlab/aydin: v0.1.15, 2022. 1

[77] Sanghyun Son, Matheus Gadelha, Yang Zhou, Zexiang Xu, Ming Lin, and Yi Zhou. Dmesh: A differentiable mesh representation. In *Advances in Neural Information Processing Systems*, pages 12035–12077. Curran Associates, Inc., 2024. 2

[78] Wang T. Michaelos M. et al. Stringer, C. Cellpose: a generalist algorithm for cellular segmentation. *Nat Methods*, 18: 100–106, 2021. 1

[79] Wim Thiels, Bart Smeets, Maxim Cuvelier, Francesca Caroti, and Rob Jelier. spheresDT/Mpacts-PiCS: cell tracking and shape retrieval in membrane-labeled embryos. *Bioinformatics*, 37(24):4851–4856, 2021. 3

[80] Virginie Uhlmann, Julien Fageot, and Michael Unser. Hermitte snakes with control of tangents. *IEEE Transactions on Image Processing*, 25(6):2803–2816, 2016. 1, 3

[81] Virginie Uhlmann, Pavan Ramdya, Ricard Delgado-Gonzalo, Richard Benton, and Michael Unser. Flylimb-tracker: An active contour based approach for leg segment tracking in unmarked, freely behaving drosophila. *PLoS One*, 12(4):e0173433, 2017. 1
[82] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Si-
moncelli. Image quality assessment: From error visibility to
structural similarity. *IEEE Transactions on Image Process-
ing*, 13(4):600–612, 2004. 7

[83] Schmidt U. Boothe T. et al. Weigert, M. Content-aware image restoration: pushing the limits of fluorescence microscopy. *Nat Methods*, 15:1090–1097, 2018. 1

[84] Veit Wiesmann, Matthias Bergler, Ralf Palmisano, Martin Prinzen, Daniela Franz, and Thomas Wittenberg. Using simulated fluorescence cell micrographs for the evaluation of cell image segmentation algorithms. *BMC bioinformatics*, 18(1):1–12, 2017. 2