# Robust Plant Cell Tracking in Fluorescence Microscopy 3D+T Series

Manuel Petit, Guillaume Cerutti, Christophe Godin, Grégoire Malandain

► To cite this version:

Manuel Petit, Guillaume Cerutti, Christophe Godin, Grégoire Malandain. Robust Plant Cell Tracking in Fluorescence Microscopy 3D+T Series. ISBI 2022 - IEEE 19th International Symposium on Biomedical Imaging, Mar 2022, Kolkata, India. pp.1-4, 10.1109/ISBI52829.2022.9761521. hal-03683119

HAL Id: hal-03683119

https://hal.inrae.fr/hal-03683119v1

Submitted on 6 Jan 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
# ROBUST PLANT CELL TRACKING IN FLUORESCENCE MICROSCOPY 3D+T SERIES

Manuel Petit*

Guillaume Cerutti*

Christophe Godin*

Grégoire Malandain†

* Laboratoire Reproduction et Développement des Plantes,
Univ Lyon, ENS de Lyon, UCB Lyon 1, CNRS, INRAE, Inria, Lyon, France
†Université Côte d'Azur, Inria, CNRS, I3S, France

## ABSTRACT

Automatic tracking of cell deformation during development using time-lapse confocal microscopy is a challenging task. In plant cell tissues, large deformations and several division cycles can occur between two consecutive time-points making the image registration and tracking procedure particularly difficult. Here, we propose an iterative approach where an initial registration transformation and cell-to-cell mapping are alternatively refined using high-confidence associations selected on the basis of a geometric context preservation score. The method, evaluated on a long time-lapse series of floral meristem, clearly demonstrates its superiority over a non-iterative approach. In addition, we show that the geometric context preservation score can be used to define a lineage quality assessment metric that makes it possible for an expert to provide locally nudges to finalize the lineage detection if necessary in a semi-automatic way.

**Index Terms—** Fluorescence microscopy, morphogenesis, cell tracking, plant, meristem

## 1. INTRODUCTION

Understanding the genetic regulation of morphogenesis in animal and plant development is a central question in developmental biology. With the improvement of microscopy imaging techniques, time-lapse series of images can be acquired and used to track labeled cells in a tissue over long periods of development. Because of the accumulation of large volume of image data, the automation of this challenging geometrical (tissue deformation) and topological (cell division) problem is attracting great attention.

The tracking methods depend on the type of cell structure labeled: the nuclei or the membrane. The nuclei tracking that can be seen as a particle tracking has been extensively studied in the 2D case with some recent 3D cases [?]? where performances and generalisation of the methods are tested each year in the Cell Tracking Challenge<fnref n="1" />. The tracking of the cell membrane has been applied mostly for the epithelial cells [2, 1] and plant cells [3, 6, 4].

In the case of plants that develop relatively slowly compared to animals, the temporal resolution of imaging systems is of the order of several hours, mainly to avoid phototoxicity. This can result in large deformations and several cycles of divisions between consecutive frames. Various methods based on local-graph matching to track cells in the shoot-apical meristem (SAM) have been proposed but limited their application to small deformation cases [4, 7, 5, 6]. In large deformation cases, Fernandez and al. developed an iterative tracking scheme where high-confidence lineage are selected and used to improve the temporal registration [3]. A limitation of the

method was the need to select manually a set of high-confidence lineage to initialize the registration procedure. Moreover, the set of high-confidence lineage can not be updated during the iterative process which can lead to error accumulation.

In this paper, we describe a robust method to track cell membrane when temporal resolution is low compared to development. Similarly to [3], the tracking procedure is embedded in an iterative scheme. Our main contribution lies in the definition and use of a tracking quality assessment (TQA) score to identify high-confidence associations and that makes it possible to automatically detect regions of low lineage quality to easily further guide the lineage detection with nudges in a semi-automatic manner.

## 2. ITERATIVE TRACKING METHOD

Let us denote $I_t$ a 3D intensity image observed at time $t$, $I_t: \Omega \subset \mathbb{Z}^3 \rightarrow \mathcal{I}$, where $\Omega$ is the image domain, and $\mathcal{I}$ is a finite set of intensities. A segmented image $J_t: \Omega \rightarrow \mathcal{L}_t$, where $\mathcal{L}_t = \{\ell_t^i\}_{i \in [1, N_t]}$ is a finite set of labels, defines corresponding cell regions $c_t^i$ in $\Omega$. Label $i=1$ denotes the image background while the other labels $\ell_t^i$ correspond to cells $c_t^i$.

Let us consider two images $I_t$ and $I_{t'}$ at times $t$ and $t' = t + \Delta t$. Tracking cells consists of finding a mapping $\mathcal{M}: \mathcal{L}_t \rightarrow \mathcal{P}(\mathcal{L}_{t'})$ that associates one label $\ell_t^i$ from $J_t$ with one or more (to account for potential divisions) labels $\ell_{t'}^j$ from $J_{t'}$.

Each iteration $n$ of the proposed tracking method consists of 3 successive steps (Fig. 1): computation of a non-linear transformation $T_{t \leftarrow t'}^n$ that superimposes $I_t$ and $I_{t'}$ (section 2.1), computation of a mapping $\mathcal{M}_n$ from $\mathcal{L}_t$ to $\mathcal{P}(\mathcal{L}_{t'})$ (section 2.2), and update of a subset $\mathcal{M}_n$ of high-confidence associations from the mapping (section 2.4).

<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Principle of the TQA-based iterative method.</figcaption></figure>

<footnote n="1">¹http://celltrackingchallenge.net/</footnote>
## 2.1. Image registration

The non-linear transformation $T_{t \leftarrow t'}^n$ is computed in two steps, first by computing an affine transformation, then by refining it by using a deformable transformation, encoded by a dense vector field. Note that all image-based registration are performed with a hierarchical block-matching strategy [8].

At the first iteration (0), the affine transformation $A_{t \leftarrow t'}^0$ is computed between images $I_t$ and $I_{t'}$, possibly by providing an initial rigid transformation to compensate for a large pose difference. Then the deformable transformation $D_{t \leftarrow t'}^0$ is computed between $I_t \circ A_{t \leftarrow t'}^0$ and $I_{t'}$. The final transformation is then $T_{t \leftarrow t'}^0 = A_{t \leftarrow t'}^0 \circ D_{t \leftarrow t'}^0$.

At other iterations $n$, the set of high-confidence associations $M_{n-1} = \{(\ell_t^i, L_t^i)\}$ is used to compute pairings between the centroid of cell $c_t^i$ and the one of $\cup_{\ell_{t'}^j \in L_{t'}^i} c_{t'}^j$. The affine transformation, $A_{t \leftarrow t'}^n$, is computed by a least squares minimization of the distance between paired centroids. Residual errors on the pairings are used to build a first dense vector field (a non-linear transformation) by propagating locally the sparse vectors that are defined by centroid pairs, that is further refined by image-based registration yielding $D_{t \leftarrow t'}^n$. The final transformation is again $T_{t \leftarrow t'}^n = A_{t \leftarrow t'}^n \circ D_{t \leftarrow t'}^n$.

## 2.2. Temporal mapping

$T_n^{t \leftarrow t'}$ allows to resample $J_t$ onto $J_{t'}$. Let us denote the resampled cells $c_t^i$ of $J_t \circ T_{t \leftarrow t'}^n$ by $T(c_t^i)$. The rationale of the temporal mapping is to associate a cell from $J_{t'}$ with the one with maximal intersection in $J_t \circ T_n^{t \leftarrow t'}$.

For this, we define a cost function $\gamma(\ell_t^i, \ell_{t'}^j)$ that estimates the association cost of a pair of labels from $\mathcal{L}_t$ and $\mathcal{L}_{t'}$:

$$ \gamma(\ell_t^i, \ell_{t'}^j) = 1 - \frac{v(T(c_t^i) \cap c_{t'}^j)}{v(c_{t'}^j)} $$

where $v(c)$ is the volume of the region c.

At iteration $n$, the temporal mapping $M_n$ is constructed by associating each label $\ell_{t'}^j$ with the label $\ell_t^i$ that minimizes $\gamma$:

$$ \ell_t^{i*} = \arg \min_{\ell_{t'}^i \in \mathcal{L}_t} \gamma(\ell_t^i, \ell_{t'}^j) \quad \text{with} \quad \gamma(\ell_t^i, \ell_{t'}^j) < 1 \quad (1) $$

Both backgrounds (labels '1' from $J_t$ and $J_{t'}$) are associated with each other exclusively. Note that a cell of $J_t$ can be associated with several cells of $J_{t'}$ but can also be associated with none of them. Likewise, a cell from $J_{t'}$ can also be not associated with any cell of $J_t$. A complete mapping of $\mathcal{L}_t$ into $\mathcal{L}_{t'}$ can be obtained by transitively grouping respectively the cells of $J_t$ and $J_{t'}$ without a correspondent cell into fake cells and by associating these two fake cells.

## 2.3. Tracking quality assessment (TQA) score

In plants, the contact between adjacent cells is conserved during growth as cells do not slide with respect to each other. This property gives us a means to assess the quality of potentially correct associations in the mapping $M_n$.

Let us denote $s_t^{i,k}$ the contact surface between the cells $c_t^i$ and $c_t^k$, $S_t^i$ the total surface of cell $c_t^i$, and $a_{i,k} = \frac{s_t^{i,k}}{S_t^i}$ for $i \neq k$ and $a_{i,i} = 0$ otherwise. We define the geometric context of a cell $c_i$ as the vector of normalized contact surfaces of each cell with its neighbors. For the image $J_t$, the geometric context $g(c_t^i)$ is thus:

$$ g(c_t^i) = (a_{i,1} \cdots a_{i,k} \cdots a_{i,N_t})^\mathrm{T} \quad (2) $$

From $M_n$ we can build a one-to-one mapping between cells $c_t^i$ of $J_t$ and super-cells $C_{t'}^i$ resulting from the aggregation of cells from $M_n(\ell_t^i)$. We define the geometric context $g(C_{t'}^i)$ of super-cells similarly to the geometric context of simple cells $g(c_t^i)$.

A TQA score $p(c_i)$ is then calculated for each cell as the distance between mapped geometric contexts:

$$ p(c_i) = 1 - \frac{\|g(c_t^i) - g(C_{t'}^i)\|_1}{2} \quad (3) $$

where $0 \le p(c_i) \le 1$. A TQA score close to 1 indicates that the distribution of the relative contact surface are well-preserved by the lineage mapping $M_n$.

## 2.4. High-confidence associations

To further refine the registration of $I_t$ with $I_{t'}$ (section 2.1), we aim at extracting a subset $M_n$ of high-confidence associations from the mapping $M_n$.

Due to the way plants grow, mapped cell volumes cannot decrease. We thus first excluded cells that show a global volume decrease (the volume of the super-cell $C_{t'}^i$ is smaller than its parent cell $c_t^i$), as well as cells $c_{t'}^j$ defined by $L_{t'}^j$ that do not form a single connected component. This defines a subset $M_n'$ of candidate cells for further high-confidence lineage detection.

At the first iteration (0), only the remaining associations with a TQA score above a threshold $p_\tau$ (set empirically to 0.85) are considered. This defines a subset of cells $c_t^i$ of $J_t$. From this subset, only the largest connected component of cells $c_t^i$ is kept to defined the first set $M_0$ of high-confidence associations.

In the next iterations $n$, additional high-confidence associations are aggregated to the connected component of cells $c_t^i$ issued from $M_{n-1}$, based both on the TQA score and the adjacency to the connected component (The adjacency of a cell $c_t^i$ to the connected component is defined as the number of cells of the connected component $c_t^i$ is adjacent with). First, we update the associations already in $M_{n-1}$. For those associations, for every cell, if the set $L_{t'}^j$ has changed, it replaces the former one in $M_{n-1}$ if the TQA score is larger. Second, we consider adding new associations to $M_{n-1}$. This is an iterative procedure, starting with $M = M_{n-1}$ that proceeds in three steps. (1) the associations of $M_n'$ are sorted in descending order according to their TQA score. Then (2) if the cell $c_t^i$ was not already in $M$, the corresponding association is added to $M$ if $c_t^i$ had an adjacency above a threshold $N_{adj}$ (typically 3), and the procedure stops if enough associations have been added. Finally (3) if not enough associations have been added, $N_{adj}$ is decreased and we go back to step (1).

To reach convergence in a reasonable amount of iterations, a minimal number of new associations (typically 5%) from $M_n'$ is required to be added to $M_{n-1}$ to form $M_n$.

# 3. RESULTS

## 3.1. Dataset and evaluation metric

The tracking method was evaluated on a series of 3D time-lapse images of the development of a floral meristem of *Arabidopsis Thaliana* [9]. This series contains 18 time-points coarsely rigidly registered and separated by about 8 h each. The floral meristem was imaged from the initiation until the stage 4 using a yellow fluorescent protein (YFP) targeted to the plasma membrane. The time-lapse
<figure><img src="image_3.png" /><figcaption>**Fig. 2.** Time-lapse series used for the evaluation of the tracking approach (FM1 in [9]). The top part displays 2D projection of the complete sequence of 3D intensity images ($\Delta t \approx 8$h). The bottom part displays some 2D projection of intensity images and mesh representation of segmented images from different developmental stages.</figcaption></figure>

is available from a public repository<fnref n="2" /> (series FM1) that contains also the segmented 3D time-lapse and an expertized lineage valid on the two outer tissue layers (L1 and L2). The time-lapse image sequence is displayed in Fig. 2.

All the images were resampled prior tracking to the average image voxel size of the sequence ie. $0.23 \times 0.23 \times 0.14 \ \mu m^3$. We also extended the expertized lineage to the third layer (L3) and detected and corrected a few expertized associations in the border regions of the images. This updated expertized lineage is available at a Gitlab repository<fnref n="3" />.

We assessed the accuracy of computed mappings $\mathcal{M}$ with the ratios of mother cells that have a correct progeny $\alpha(\mathcal{M}) = \frac{M_C(\mathcal{M})}{M_T}$, and the ratio of daughter cells that have the correct mother $\beta(\mathcal{M}) = \frac{D_C(\mathcal{M})}{D_T}$. $M_C(\mathcal{M})$ denotes the number of correct mother cell associations in $\mathcal{M}$ (i.e. mother cells associated with all the expertized daughter cells), $D_C(\mathcal{M})$ the number of daughter cells correctly associated with their expertized mother cell, and $M_T$ (resp. $D_T$), the total number of expertized mother (resp. daughter) cells.

## 3.2. Oneshot vs. iterative tracking

To assess the improvement of computed lineages due to the above iterative method, we compared its final results to that obtained after one iteration only (one-shot method). For this, we selected pairs of time points separated by either a short time-step ($\Delta t = [4, 10]h, N = 17$) or long time-step ($\Delta t = [12, 18]h, N = 16$). The long time-step pairs were obtained by taking every second images instead of consecutive images in the time-lapse series. We then computed the accuracy of identified lineages for each time-step class Fig. 3.

For short time-steps, the overall median accuracy, already quite high for the one-shot method, was slightly increased from 0.965 to 0.974 by the iterative method. However, the overall median accuracy was clearly increased (from 0.882 to 0.913) for long time-steps (with an improvement particularly visible in the L1 layer, from 0.876

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Quantitative comparison of the non-iterative approach (oneshot) and iterative approach. Both methods are evaluated on all the available pairs of time points ($N = 17$ for short time-step; $N = 16$ for long time-step). (a) $\alpha(\mathcal{M}) = \frac{M_C(\mathcal{M})}{M_T}$ (ratio of mother cells with correct progeny) boxplots: from left to right, results for all the layers, then results for the different layers (L1, L2, L3). (b) $\alpha(\mathcal{M})$ and $\beta(\mathcal{M}) = \frac{D_C(\mathcal{M})}{D_T}$ (radio of daughter cells with correct ancestor) for 8 long time-step cases (border cells are excluded from the analysis). The number of correct associations is indicated at the top of each bar and $N$ denotes the total number of expertized associations.</figcaption></figure>

to 0.933). Interestingly, for both time steps, the dispersion of the accuracy was markedly reduced in the iterative method, showing that the results were altogether more consistent and biologically meaningful with the iterative method.

We noted that for short time-steps most of the predicted association errors happened at the border of the segmented images (cells that intersects the image border). To assess the tracking errors in the zone of biological interest proper, we discarded errors in the border regions (see the Gitlab repository<fnref n="3" />) in Fig. 3b. The iterative method reaches very high accuracy scores on average ($\ge 0.95$) in these key regions.

More generally, our experiments demonstrate that both oneshot and iterative tracking methods perform well when the co-registration of $I_t$ and $I_{t'}$ has succeeded. However, in case of a poor initial registration, the iterative tracking method performs much better since it is able to improve the initial transformation: this is particularly demonstrative when the flower bud grows rapidly.

<footnote n="2">²https://www/repository.cam.ac.uk/handle/1810/318119</footnote>

<footnote n="3">³https://gitlab.inria.fr/mosaic/publications/isbi_2022</footnote>
<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Visualization of the detection and correction of potential tracking errors using nudges on pair of time-points $t=112$ h (mother image) and $t'=128$ h (daughter image). First column corresponds to the expertized mapping expected between mother (top) and daughter (bottom) images. Second column corresponds to the TQA score map of the iterative mapping (top) and its corresponding daughter image (bottom). For each potential error regions found in the TQA map (black rectangles), a pair of landmarks is manually set. Third column corresponds to the TQA score map of the mapping obtained after the use of nudges (top) and its associated daughter image (bottom).</figcaption></figure>

### 3.3. Detection and correction of potential tracking errors

The TQA score is used in the iterative process to automatically select the high-confidence associations. Thus, regions containing potential tracking errors can also be detected. This information can help to speed up the visual validation of the mapping and point out potential errors. Moreover, in the case of errors (unavoidable for large deformations, or in case of segmentation errors), a single pair of landmark in the region of interest can be enough to correct locally the deformation. Briefly, pair of landmarks are used to defined a deformable transformation $D_{t'-t}^a$ (similarly as section 2.1) that is composed to the current transformation $T_{t'-t}^n$ and refined further by image-based registration yielding $T_{t'-t'}^a$. This last transformation is then used to obtain a mapping $M^a$ following the procedure described in section 2.2. Fig 4 presents the improvement simultaneously observed in the TQA map score and the mapping when providing single pair of landmarks per potential error regions. In average, for one pair of landmarks about 8 additional correct associations were recovered on the pair of time-points displayed on Fig.3b.

## 4. CONCLUSION

In this paper, we have proposed an iterative method to track plant cells in 3d+t sequence of images when large deformations occurs between two time-points. The capability of the tracking approach to improve an initial tracking has been demonstrated using a proposed geometric preservation context score. The improvement of the lineage accuracy were particularly visible when the flower bud grows rapidly and sepals appears, a critical step where important non-linear deformations arises. Moreover, we demonstrate the great interest of the proposed geometric preservation context score enabling the detection of high-confidence associations for the automatic tracking and potential error regions for semi-automatic validation and correction. For an expert user, this score can help to speed up the evaluation

of a given lineage and define locally nudges able to correct the possible last errors. Future research directions are to validate the method on other sequences of images and extend the tracking capability to detect segmentation errors using alternative temporal mapping approach.

## 5. COMPLIANCE WITH ETHICAL STANDARDS

This is a study for which no ethical approval was required.

## 6. ACKNOWLEDGMENTS

This work has been funded by the Inria project Naviscope (Image-guided Navigation and Visualization of large data sets in live cell imaging and microSCOPy).

## 7. REFERENCES

[1] A. Alladin, L. Chaible, L. G. Del Valle, R. Sabine, M. Loeschinger, M. Wachsmuth, J.-K. Hériché, C. Tischer, and M. Jechlinger. Tracking cells in epithelial acini by light sheet microscopy reveals proximity effects in breast cancer initiation. *Elife*, 9:e54066, 2020.

[2] D. L. Farrell, O. Weitz, M. O. Magnasco, and J. A. Zallen. Segga: a toolset for rapid automated analysis of epithelial cell polarity and dynamics. *Development*, 144(9):1725–1734, 2017.

[3] R. Fernandez, P. Das, V. Mirabet, E. Moscardi, J. Traas, J.-L. Verdeil, G. Malandain, and C. Godin. Imaging plant growth in 4d: robust tissue reconstruction and lineaging at cell resolution. *Nature methods*, 7(7):547–553, 2010.

[4] W. Jiang, L. Wu, S. Liu, and M. Liu. Cnn-based two-stage cell segmentation improves plant cell tracking. *Pattern Recognition Letters*, 128:311–317, 2019.

[5] M. Liu, Y. He, Y. Wei, and P. Xiang. Plant cell tracking using kalman filter based local graph matching. *Image and Vision Computing*, 60:154–161, 2017.

[6] M. Liu, Y. Liu, W. Qian, and Y. Wang. Deepseed local graph matching for densely packed cells tracking. *IEEE/ACM Transactions on Computational Biology and Bioinformatics*, 18(03):1060–1069, 2021.

[7] M. Liu, P. Xiang, and G. Liu. Robust plant cell tracking using local spatio-temporal context. *Neurocomputing*, 208:309–314, 2016.

[8] S. Ourselin, A. Roche, S. Prima, and N. Ayache. Block matching: A general framework to improve robustness of rigid registration of medical images. In *International Conference on Medical Image Computing And Computer-Assisted Intervention*, pages 557–566. Springer, 2000.

[9] Y. Refahi, A. Zardilis, G. Michelin, R. Wightman, B. Leggio, J. Legrand, E. Faure, L. Vachez, A. Armezzani, A.-E. Risson, et al. A multiscale analysis of early flower development in arabidopsis provides an integrated view of molecular regulation and growth control. *Developmental Cell*, 56(4):540–556, 2021.