<figure><img src="image_1.png" /></figure>

ifip

International Federation for Information Processing

This document is the original author manuscript of a paper submitted to an IFIP
conference proceedings or other IFIP publication by Springer Nature. As such, there
may be some differences in the official published version of the paper. Such
differences, if any, are usually due to reformatting during preparation for publication or
minor corrections made by the author(s) during final proofreading of the publication
manuscript.
Visualizing Cell Motility Patterns
from Time Lapse Videos with Interactive
2D Maps Generated with Deep
Autoencoders

Ana González¹ ID, José María Enguita¹ ID, Ignacio Díaz¹(✉) ID, Diego García¹ ID,
Abel Alberto Cuadrado¹ ID, Nuria Valdés² ID, and María D. Chiara³,⁴ ID

¹ Department of Electrical Engineering, University of Oviedo, Gijón 33204, Spain
{jmenguita,idiaz,garciaperdiego,cuadradoabel}@uniovi.es

² Department of Internal Medicine, Section of Endocrinology and Nutrition, Hospital Universitario de Cabueñes, Gijón 33204, Spain
nvaldes@fis.hca.es

³ Institute of Sanitary Research of the Principado de Asturias, Hospital Universitario Central de Asturias, Oviedo 33011, Spain
mdchiara.uo@uniovi.es

⁴ CIBERONC (Network of Biomedical Research in Cancer), Madrid 28029, Spain

**Abstract.** Cell motility, the ability of cells to move, is crucial in a wide range of biological processes; for instance, in cancer, it is directly related to metastasis. However, it is a complex phenomenon which is not well-understood yet, and studies are mainly done by human observation, which is subjective and error-prone. We intend to provide an automated mechanism to analyze the movement patterns that occur in in-vitro cell cultures, which can be registered by time lapse microscopy. Our approach, which is still a work in progress, utilizes an interactive 2D map that organizes motility patterns based on their similarity, enabling exploratory analysis. We extract the velocity fields that represent the cell displacements between consecutive frames and use a deep convolutional autoencoder to project a characterization of short video sequences of smaller parts of the original videos into a 10D latent space. The samples (small videos) are visualized in a 2D map using the *Uniform Manifold Approximation and Projection* (UMAP). The possibilities and extent of our method are showcased through a small interactive application that allows to explore all the types of cell motility patterns present in the training videos on a 2D map.

**Keywords:** Deep autencoders · Cell motility analysis · Deep learning · UMAP visualization · Cancer metastasis

# 1 Introduction

Cell motility refers to the ability of cells to move spontaneously and independently, and represents a crucial dynamic process in a wide range of biological
processes. In cancer, this movement is related to cell survival and, therefore, to metastasis, which currently represents one of the great challenges in the clinical treatment of cancer. However, cell motility is a complex phenomenon, affected by the physiological context, cell type, cell morphology and cell-cell interactions. Moreover, cells can move in amoeboid, mesenchymal or epithelial modes, as individuals or in groups, etc. and can even dynamically switch between different modes in response to changing environments [22]. There is therefore great interest in studying the mechanisms underlying all types of cell motility, with the ultimate goal of identifying therapies that increase the motility of beneficial cells and block the spread of harmful cells [21].

In this context, advances in microscopy have played a key role in the study of cellular processes, as they have facilitated the acquisition of images and videos of the cell cultures of interest, making it possible to analyze their evolution over time. As a result, large datasets are generated, the exploration of which could provide intuition about new hypotheses or lines of research in the study of cell motility. However, the manual exploration of these datasets —which may contain hundreds, thousands or even millions of images— is very time-consuming for the researcher, for whom the task of finding relationships of interest in such a vast amount of data is enormously complex.

In this article, which reflects an ongoing work, we propose to explore a dataset of cell test videos by projecting them into a low-dimensional space using a combination of a deep autoencoder and the *Uniform Manifold Approximation and Projection* (UMAP) [17]. As shown below, the results of the research have demonstrated that the generated 2D map allows the identification of different patterns of cell movement, thus providing a preliminary assessment of the working samples, which represents a valuable starting point for further analysis. These results also demonstrate the potential for the qualitative characterization of cell movement, which can be potentially associated to underlying biological processes related to cancer or other conditions with biomedical relevance, thus establishing an interesting line of future work.

# 2 Methods and Techniques

## 2.1 Time Lapse Microscope Videos

The dataset consists of five cell-culture videos, characterized by very low — almost non-existent— levels of succinate dehydrogenase B (SDHB). Reduced expression or activity of SDHB, usually as a consequence of mutations in the gene that encodes it, has been described in the literature as being associated with the pathogenesis of numerous renal carcinomas [6, 7, 14, 28, 29]. There is therefore great interest in the study of this protein, in search of a better understanding of its involvement in carcinogenesis and the progression of this type of tumor. Cell movement could play a key role in the development of these processes, so we propose an exploration of working videos focused on the analysis of cell motility.
## 2.2 Video Preprocessing: Velocity Fields

Processing videos for valuable information about cell movement is a complex task. Therefore, data pre-processing is recommended to extract relevant information from the videos to feed the deep model, thus simplifying —and guiding— its learning. In particular, we calculated the velocity field of the cells, since it provides a detailed description of their motion and is a common approach in the analysis of cell motility [3, 13]. For this purpose, Gunnar Farnebäck's optical flow algorithm [8], available in the OpenCV-Python library, was used<fnref n="1" />.

It is worth noting that optical flow algorithms have only recently started to be used in the cellular domain, but they have already proven to be able to provide more accurate and robust velocity fields than other state-of-the-art techniques, such as those based on particle image velocimetry (Particle Image Velocimetry, PIV) [9, 26], or MATLAB™ imregdemons algorithm [23, 25]. “Video 1” (see appendix with videos) shows a visualisation of the velocity field obtained for one of the working videos.

As shown in Table 1a, the starting dataset contains 5 videos, of 539 frames each that, in turn, have a size of 2000×2000 pixels, where for each pixel the gray level is known. After the calculation of the velocity fields, a new set is available (Table 1b), where for each pixel, we have four variables: the horizontal and vertical components of the velocity (u, v) and the position of the pixel in the image (x, y). The velocity fields were then subjected to post-processing, which, as can be seen in Table 1c, modified the number of frames as well as their size. Based on the recommendation of the research group that provided the data<fnref n="2" />, the first 360 frames of each video were retained for the study, while the remaining frames were discarded due to their high cell density and lack of representativeness of the process's state. Secondly, the fields were decimated (with square windows of 4 × 4 size and without overlapping), in order to reduce the size of the set and thus facilitate its processing. Also, the edges (top and bottom) of the fields were discarded, as they correspond to the strips (top and bottom) of the frames, which do not contain any cellular information. As a result, the field of each frame was reduced from a size of 2000×2000 to one of 355×500.

## 2.3 Video Feature Extraction

We performed an additional feature extraction process to obtain a representative descriptor of the velocity fields. In particular, we used the *Histogram of Oriented Optical Flow (HOOF)* technique [4], shown in Fig. 1. These histograms have been shown in the literature to have great potential as movement descriptors in a variety of applications [5, 15, 24], also with recent uses in the study of cell

<footnote n="1">¹ The algorithm was implemented with *Object Tracking* of OpenCV-Python, using the following parameters: *pyr_scale* = 0.5, *levels* = 3, *winsize* = 60, *iterations* = 3, *poly_n* = 5, *polysigma* = 1.1, *flags* = cv2.OPTFLOW_FARNEBACK_GAUSSIAN.</footnote>

<footnote n="2">² Head and Neck research group from the Instituto de Investigación Sanitaria del Principado de Asturias (ISPA, https://www.ispasturias.es).</footnote>
<figure><img src="image_2.png" /><figcaption>**Fig. 1.** Example of a HOOF for a working window. A window is shown in (a), along with its corresponding velocity field. In (b) the HOOF of the velocity field is shown with different representations: in terms of a polar histogram (b.1) and in terms of a linear histogram (b.2). In this example a HOOF of 16 bins was generated.</figcaption></figure>

motility [10]. In our case, we divided the working frames into 24 windows —of size 75×75 and with no overlapping— and we calculated a 16-element HOOF for each of them. The dimensions of the resulting dataset are given in Table 1d. It must be noted that other descriptors of the vector fields —such as classical rotational or divergence operators— could be used and will be subject of future work.

Importantly, the samples in this dataset contain valuable information about the displacement of cells within each window. However, the HOOF information is limited to the displacement experienced between two consecutive frames. In order to obtain samples with a greater temporal context for the evolution of the cells, we concatenated —for each window— the current HOOF with the HOOFs of the next 15 frames, resulting in the final working package (Table 1e) consisting of 41280 samples —5×344×24 samples— of size 16×16.

Finally, this dataset was normalized using a *min-max* scale [19] with range [0, 1], divided randomly into two subsets: training (70%, 28896 samples) and test (30%, 12384 samples), for the training and evaluation of the deep autoencoder presented in Sect. 2.4.

## 2.4 2D Map of the Cell Motility Dataset

Each of the 41,280 samples describes locally the type of cellular movement contained in a small video of 16 frames of 75 × 75 pixels each, using a 16 × 16 matrix of HOOF of features (16 HOOF descriptors × 16 frames), equivalent to 256 dimensions. To visually represent the types of movement of the samples, organized spatially by similarities, a dimensionality reduction method is proposed to project the samples onto a 2D space for visualization.

For that purpose, a two-step dimension reduction was used: first, the original dimension of the dataset was reduced from 256D to a latent space with a lower dimensionality by means of a deep autoencoder. Next, the data were
<table><caption>**Table 1.** Cell motility dataset (a) and datasets derived from its preprocessing (b, c, d, e). The # of samples of each dataset is expressed as: a) # of videos × # of frames × frame width × frame height; b, c) # of videos × # of frames × field width × field height; d, e) # of videos × # of frames × # of windows.</caption><thead><tr><td>Dataset</td><td># of samples</td><td>Size of the samples</td></tr></thead><tbody><tr><td>a) Starting data</td><td>(5×539×2000×2000)</td><td>(1)</td></tr><tr><td>b) Velocity fields</td><td>(5×539×2000×2000)</td><td>(4)</td></tr><tr><td>c) Post-processed velocity fields</td><td>(5×360×355×500)</td><td>(4)</td></tr><tr><td>d) HOOF</td><td>(5×360×24)</td><td>(16)</td></tr><tr><td>e) HOOF temporal</td><td>(5×344×24)</td><td>(16×16)</td></tr></tbody></table>

projected from the latent space to a 2D visualization space using a neighbor embedding technique. As shown in other research works [2, 16], the success of this combination is due to the fact that it exploits: on one hand, the ability of deep autoencoders to generate low-dimensional, disentangled latent representations of the data; and, on the other hand, the ability of the neighbor embedding techniques to provide compact, visualizable (2D) representations of such latent representations with good separability properties, which makes them especially useful in clustering applications. In this case, we used the UMAP technique, as proposed in other works in the literature [1, 18, 27]. In line with these works, the results obtained demonstrate that, even though the high dimensionality of the data makes it difficult to capture its structure in a two-dimensional space, it is possible to generate quality 2D maps with meaning about the processes thanks to the combination of both DR techniques.

After trying several dimensionalities for the latent space, the autoencoder training showed poorer reconstruction results when trained with dimensionalities below 10D, so we decided to keep this number of dimensions for the latent space, followed by the 2D projection step.

The autoencoder was trained using the gradient descent algorithm [20] combined with the ADAM optimizer [11]. The number of epochs, the size of the *mini-batch* and the architecture are detailed in Table 2. This table shows that the autoencoder trained with the cell motility dataset contains convolutional and subsampling layers as recommended in the literature for 2D data processing [12]. Regarding the activation functions, the ReLU function was used in all the layers except in the output layer and in the bottleneck of the model, where a linear activation function was used. For the training of the UMAP model, after testing several configurations, a number of 5 neighbors and a minimum distance of 0.5 were chosen since they were found to provide a good distribution of the different types of cell motility.

Regarding the choice of hyperparameters for the autoencoder (number of layers, number of neurons in the layers, number of epochs, size of the “mini-batch”, etc.), we ran different ranges of experiments and chosen those hyperparameters with which the model showed the lowest reconstruction error. The reconstruc-
<table><caption>**Table 2.** Architecture of the *deep autoencoder*. All the layers are dense, except the following: Conv2D (2D convolutional layer), MaxPooling2D (subsampling by max value), Flatten (unidimensional conversion layer), Reshape (multidimensional conversion layer), Conv2DTranspose (deconvolution layer). The convolutional and subsampling layers were configured with a null (*padding*) and a (*stride*) of 1, except in the first deconvolution layer, for which a stride of 2 was used.</caption><thead><tr><th rowspan="2"># of epochs</th><th rowspan="2">Size of mini-batch</th><th rowspan="2"># of layers</th><th colspan="3"># neurons in layers</th></tr><tr><th>Encoder</th><th>Bottleneck</th><th>Decoder</th></tr></thead><tbody><tr><td>600</td><td>1000</td><td>15</td><td>(16×16×1,<br>14×14×8 Conv2D,<br>12×12×4 Conv2D,<br>10×10×2 Conv2D,<br>5×5×2 MaxPooling2D,<br>50 Flatten, 20)</td><td>(10)</td><td>(20, 50,<br>5×5×2 Reshape,<br>12×12×2 Conv2DTranspose,<br>14×14×4 Conv2DTranspose,<br>16×16×8 Conv2DTranspose,<br>16×16×1 Conv2D)</td></tr></tbody></table>

tion error has been evaluated on the test subset, while the model was fed with the training subset.

## 2.5 Interactive Visualization

In order to demonstrate the capability of the approach proposed in this article for distributing and clustering different cellular movement patterns across a 2D map, the proposed processing was integrated into an interactive application, solely for demonstrative purposes, that is shown in Fig. 3 and can be watched in “video 2” of the appendix of Sect. 5. The application allows for the representation of three views for each sample: 1) its projection on the 2D map; 2) a representative heatmap of 16 × 16 pixels with the value of 16 HOOF descriptors for the 16 frames; and 3) the original video clip. As shown in “video 2”, using this application the different movement patterns on the map, including translation, rotation, crawling, among others, can be identified and are spatially organized along the map in a coherent manner.

# 3 Results and Discussion

Figure 2 shows the map obtained after a dimensionality reduction done with the proposed mapping obtained with the combination of the deep autoencoder and UMAP. The combination of both approaches provides a regularized representation of the data, which also presents a good separability of the different cell motility patterns.

As shown in Fig. 4, the exploration of the map allows to find different patterns of movement in the data, represented in color in the figure (it should be noted that the map has not been fully explored yet, given the large volume of samples to be handled; unexplored samples are shown in gray color).

In detail, zones with absence or presence of cells, zones where rotation or displacement movements predominate, zones of interaction between cells, etc. were identified. Therefore, the map provided a compact and meaningful representation of the data, whose semantics could be explored interactively. Finally,
<figure><img src="image_3.png" /><figcaption>Fig. 2. 2D map of the cell motility types using a deep autoencoder + UMAP</figcaption></figure>

<figure><img src="image_4.png" /><figcaption>**Fig. 3.** Interactive visualization of the cell motility map. The primary visualization consists of a 2D scatter plot that displays the projected samples (mini videos). On the right side of the plot, there is a heatmap that shows the HOOF description of the currently selected sample. Below the heatmap, there is an animated view of the selected sample.</figcaption></figure>
an overall animation of the map is shown in “video 3”, where several example samples were included for each of the motility patterns identified.

<figure><img src="image_5.png" /><figcaption>**Fig. 4.** Cell motility map labeled by movement patterns (samples that have not been labeled are presented in gray).</figcaption></figure>

# 4 Conclusions

In this article, we have introduced a novel approach for automatically detecting relevant patterns of movement in cell cultures using deep autoencoders. Although this work represents a preliminary study, the results demonstrate significant potential. Notably, the interactive visualization of the latent space illustrates the autoencoder’s ability to classify different movement patterns and represents a robust preliminary step in their automated processing.

The 2D map of the data not only serves as a valuable tool for exploring the dataset, but also provides evidence for the potential of cell motility analysis approaches in investigating cancer processes. In particular, the resulting map enables the identification of potential avenues for future research into the mechanisms linking the SDHB protein to the development and evolution of renal carcinomas. For instance, the map permits the analysis of behavioral differences between cell cultures exhibiting low levels of SDHB (i.e., the working dataset) and other cultures of varying nature (e.g., control cultures or cultures with normal SDHB levels) by projecting them onto the map.

Furthermore, the identification of distinct motility zones on the 2D map also provides an opportunity to explore other lines of interest for future research
related to unsupervised sample labeling. Such labeling could enable a compre-
hensive analysis of the dataset, by excluding samples with no relevant informa-
tion (i.e., samples without cells) or by focusing exclusively on specific patterns
of interest present on the map.

However, certain limitations exist in this study. Firstly, the analyzed videos pertain to a specific problem, and images in other contexts may differ significantly. Although we do not anticipate this to pose a challenge for the adaptation of the described technique, no tests have been performed to validate this assumption. Secondly, the use of HOOF for extracting movement features in each mini-video also presents certain limitations, which could explain why some patterns are not classified correctly. Notably, HOOF does not account well for patterns with local symmetries in the directionality, which suggests that alternative methods could potentially enhance the results. Exploring the applicability of other descriptors is an interesting line of future work.

As a final note, although this article has focused on a specific problem, the applications of this technique can extend to numerous different fields in which the analysis of cell motility is critical.

# 5 Appendix: Videos

video 1. https://www.youtube.com/watch?v=gisz0Px4k.8 → Velocity fields

video 2. https://www.youtube.com/watch?v=zVWGxsjtvsg → Interactive visualization of cell motility dataset

video 3. https://www.youtube.com/watch?v=dZoB_2Wi9Co → Patterns of movement in the cell motility map

**Acknowledgments.** This work was supported by the Ministerio de Ciencia e Inno-
vación / Agencia Estatal de Investigación (MCIN/AEI/ 10.13039/501100011033) grant
[PID2020-115401GB-I00]. The authors would also like to thank the financial support
provided by the Principado de Asturias government through the predoctoral grant
“Severo Ochoa”.

References

1. Ali, M., Jones, M.W., Xie, X., Williams, M.: Timecluster: dimension reduction applied to temporal data for visual analytics. Vis. Comput. **35**(6), 1013–1026 (2019)

2. Allaoui, M., Aissa, N.E.H.S.B., Belghith, A.B., Kherfi, M.L.: A machine learning-based tool for exploring covid-19 scientific literature. In: 2021 International Conference on Recent Advances in Mathematics and Informatics (ICRAMI), pp. 1–7. IEEE (2021)

3. Camley, B.A., Rappel, W.J.: Physical models of collective cell motility: from cell to tissue. J. Phys. D Appl. Phys. **50**(11), 113002 (2017)

4. Chaudhry, R., Ravichandran, A., Hager, G., Vidal, R.: Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions. In: 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 1932–1939. IEEE (2009)
5. Colque, R.V.H.M., Caetano, C., de Andrade, M.T.L., Schwartz, W.R.: Histograms of optical flow orientation and magnitude and entropy to detect anomalous events in videos. IEEE Trans. Circuits Syst. Video Technol. **27**(3), 673–682 (2016)

6. Cornejo, K.M., et al.: Succinate dehydrogenase B: a new prognostic biomarker in clear cell renal cell carcinoma. Hum. Pathol. **46**(6), 820–826 (2015)

7. Fang, Z., Sun, Q., Yang, H., Zheng, J.: SDHB suppresses the tumorigenesis and development of ccRCC by inhibiting glycolysis. Front. Oncol. **11**, 639408 (2021)

8. Farnebäck, G.: Two-frame motion estimation based on polynomial expansion. In: Bigun, J., Gustavsson, T. (eds.) SCIA 2003. LNCS, vol. 2749, pp. 363–370. Springer, Heidelberg (2003). https://doi.org/10.1007/3-540-45103-X_50

9. Hoshikawa, E., et al.: Cells/colony motion of oral keratinocytes determined by non-invasive and quantitative measurement using optical flow predicts epithelial regenerative capacity. Sci. Rep. **11**(1), 1–12 (2021)

10. Huang, Y., Hao, L., Li, H., Liu, Z., Wang, P.: Quantitative analysis of intracellular motility based on optical flow model. J. Healthcare Eng. **2017**, 1848314 (2017)

11. Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 (2014)

12. Krizhevsky, A., Sutskever, I., Hinton, G.E.: ImageNet classification with deep convolutional neural networks. Advances in Neural Information Processing Systems 25 (2012)

13. Ladoux, B., Mège, R.M.: Mechanobiology of collective cell behaviours. Nat. Rev. Mol. Cell Biol. **18**(12), 743–757 (2017)

14. Linehan, W.M., et al.: The metabolic basis of kidney cancer. Cancer Discov. **9**(8), 1006–1021 (2019)

15. Liu, Y.J., Zhang, J.K., Yan, W.J., Wang, S.J., Zhao, G., Fu, X.: A main directional mean optical flow feature for spontaneous micro-expression recognition. IEEE Trans. Affect. Comput. **7**(4), 299–310 (2015)

16. McConville, R., Santos-Rodriguez, R., Piechocki, R.J., Craddock, I.: N2d:(not too) deep clustering via clustering the local manifold of an autoencoded embedding. In: 2020 25th International Conference on Pattern Recognition (ICPR), pp. 5145–5152. IEEE (2021)

17. McInnes, L., Healy, J., Melville, J.: UMAP: uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv:1802.03426 (2018)

18. Morehead, A., Chantapakul, W., Cheng, J.: Semi-supervised graph learning meets dimensionality reduction. arXiv preprint arXiv:2203.12522 (2022)

19. Patterson, J., Gibson, A.: Deep learning: a practitioner's approach. O'Reilly Media, Inc. (2017)

20. Ruder, S.: An overview of gradient descent optimization algorithms. arXiv preprint arXiv:1609.04747 (2016)

21. Stuelten, C.H., Parent, C.A., Montell, D.J.: Cell motility in cancer invasion and metastasis: insights from simple model organisms. Nat. Rev. Cancer **18**(5), 296–312 (2018)

22. Te Boekhorst, V., Preziosi, L., Friedl, P.: Plasticity of cell migration in vivo and in silico. Annu. Rev. Cell Dev. Biol. **32**(1), 491–526 (2016)

23. Thirion, J.P.: Image matching as a diffusion process: an analogy with Maxwell's demons. Med. Image Anal. **2**(3), 243–260 (1998). https://doi.org/10.1016/S1361-8415(98)80022-4

24. Verburg, M., Menkovski, V.: Micro-expression detection in long videos using optical flow and recurrent neural networks. In: 2019 14th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2019), pp. 1–6. IEEE (2019)
25. Vercauteren, T., Pennec, X., Merchant, A., Ayache, N.: Diffeomorphic demons: Efficient non-parametric image registration. NeuroImage **45**(1, Supplement 1), S61–S72 (2009). https://doi.org/10.1016/j.neuroimage.2008.10.040. https://www.sciencedirect.com/science/article/pii/S1053811908011683, mathematics in Brain Imaging

26. Vig, D.K., Hamby, A.E., Wolgemuth, C.W.: On the quantification of cellular velocity fields. Biophys. J. **110**(7), 1469–1475 (2016)

27. Wang, Y., Yu, Z., Wang, Z.: A temporal clustering method fusing deep convolutional autoencoders and dimensionality reduction methods and its application in air quality visualization. Chemom. Intell. Lab. Syst. **227**, 104607 (2022)

28. Yang, J., et al.: Functional deficiency of succinate dehydrogenase promotes tumorigenesis and development of clear cell renal cell carcinoma through weakening of ferroptosis. Bioengineered **13**(4), 11187–11207 (2022)

29. Yong, C., Stewart, G.D., Frezza, C.: Oncometabolites in renal cancer. Nat. Rev. Nephrol. **16**(3), 156–172 (2020)