This paper introduces an intersection of computer vision, organoid biology, and artificial intelligence by naming it Organoid Computer Vision (OCV). OCV is emerging as an interdisciplinary field that leverages advanced computational techniques to analyze and interpret organoid imaging data. This survey provides a comprehensive overview of the current state of OCV , exploring its foundational principles, methodologies, and applications. It begins by defining the scope of OCV and highlighting its significance in advancing both organoid research and computer vision techniques. Key contri- butions from recent studies are analyzed, emphasizing innovations in imaging technologies, data processing algorithms, and machine learning models tailored for organoid analysis. Additionally, we discuss the challenges and limitations faced by researchers in this field, including issues related to data quality, computational complexity, and the integration of biological insights. Looking forward, we outline potential future directions for OCV research, including developing more sophisticated models, applying OCV in personalized medicine, and the ethical considerations of using computer vision in organoid research contexts. This survey aims to provide a valuable resource for researchers and practitioners, fostering further advancements and interdisciplinary collaboration in the field of Organoid Computer Vision. Keywords Organoid Computer Vision, Organoid Imaging Technologies, ·Organoid Intelligence 1 Introduction In the last decade, the fields of computer vision, stem cell research, and bio-intelligence have witnessed remarkable advancements [ 2,3]. Computer vision has seen tremendous progress in image recognition, segmentation, and analysis, driven by innovations in deep learning and artificial intelligence. Concurrently, stem cell research, particularly the development of induced pluripotent stem cells (iPSCs) [ 4] and organoid technology, has revolutionized our ability to model human development and disease in vitro [5]. Bio-intelligence and organoid intelligence have also emerged, exploring the integration of biological systems with computational frameworks to enhance our understanding of complex biological processes [6]. Organoid research [ 7], a relatively new concept, has been closely linked with 3D cell culture, stem cell research, and tissue engineering. Despite its recent emergence, there have already been various debates regarding its definition, standards, ethics, and scope [ 1]. An organoid is a self-organized 3D tissue (a miniature and simplified version of an organ) created in vitro, designed to replicate the essential functional, structural, and biological complexities of the original organ [ 2]. To successfully establish organoid-based cultures, we need to consider several key components: the types of cells used, soluble factors that guide their growth, the extracellular matrix (ECM) [ 8] they grow in, and the physical cues they are exposed to. These elements must work together seamlessly. The relevant cell types include adult stem cells (ASCs) [ 9], cancer stem cells (CSCs)[ 10], induced pluripotent stem cells (iPSCs), and tissue-derived cells (TDCs)[ 11]. The soluble factors include substances like fibroblast growth factor (FGF) [ 12]. Additionally, organ-on-a-chip (OoC) technology [ 13], which mimics the environment of organs, plays a crucial role in the process. Advanced imaging and computational techniques are essential to analyze these organoids. Both stem cell and organoid APREPRINT - SEPTEMBER 7, 2024 research heavily depend on imaging and microscopic techniques to capture detailed visual data of the organoids. This reliance on imaging means sophisticated image processing, analysis, and computer vision techniques are necessary to manage the vast amounts of data generated. Computer vision is critical because it helps to automate tasks that would otherwise be manually intensive and error- prone [ 14]. These tasks include monitoring the development and changes in organoids, detecting specific features or abnormalities, tracking the growth and movement of cells within the organoids, and controlling experimental conditions [15]. By automating these processes, researchers can achieve more accurate and efficient analysis, reducing the burden of manual labour and minimizing human error. Therefore, integrating advanced computational techniques with organoid research is crucial for studying organoids’ intricate and dynamic nature. This integration ensures the long-term success of these fields by enabling high-throughput, precise, and scalable analysis, which is vital for advancing our understanding of human biology and disease and developing new therapies and medical applications. Figure 1: Scope of Organoid Computer Vision Despite these advances, a gap remains in the systematic integration of computer vision and organoid research. This merger holds revolutionary potential but has lacked a cohesive framework and dedicated research discipline to fully realise its benefits. While some existing work [ 16] and surveys address image analysis for organoids, they often describe it as a simple pre-processing or post-processing tool with limited applicability rather than harnessing the full potential of robust computer vision techniques in tandem with organoid development. The motivation for this paper arises from the need to address the growing complexity and volume of imaging data generated by organoid research. Current manual methods of data analysis are not only labour-intensive but also prone to human error, limiting the scalability and accuracy of research outcomes. Additionally, advancements in high-resolution and 3D imaging technologies, while providing unprecedented detail and insights, require sophisticated computational techniques to exploit their potential fully. This paper seeks to bridge this gap, creating a unified framework that leverages computer vision to enhance the analysis and interpretation of organoid imaging data [ 17], thereby driving forward both fields. This paper introduces the Organoid Computer Vision (OCV) concept to address this gap and streamline research efforts. OCV represents the intersection of computer vision, organoid biology, and artificial intelligence, aiming to leverage advanced computational techniques to analyze and interpret organoid imaging data. By naming and establishing Organoid Computer Vision (OCV) as a distinct discipline, this paper seeks to survey existing work comprehensively, highlight significant contributions, and discuss the challenges and future directions in this field. We aim to encourage the research community to explore and expand upon this newly defined discipline. Therefore, the significance of this paper lies in its potential to establish Organoid Computer Vision (OCV) as a revolutionary interdisciplinary field. By automating the analysis of complex imaging data, OCV promises to significantly enhance the accuracy and efficiency of organoid research, facilitating high-throughput screening and large-scale studies. This integration accelerates the development of new therapies and personalized medicine and provides deeper insights into human biology and disease mechanisms. Moreover, by formalizing OCV , this paper fosters interdisciplinary collaboration, bringing together computer scientists, biologists, and medical researchers to drive innovation and advance the frontiers of computer vision and organoid research. Establishing this field as a distinct discipline encourages further exploration and development, setting the stage for significant scientific and medical breakthroughs. 2 APREPRINT - SEPTEMBER 7, 2024 1.1 Definition and Scope of Organoid Computer Vision Organoid Computer Vision (OCV) is an interdisciplinary field combining computer vision, organoid biology, and artificial intelligence to analyze and interpret imaging data from organoid models. The scope of OCV encompasses several key areas: OCV aims to overcome the limitations of manual and simplistic image analysis techniques by leveraging the full potential of computer vision to advance our understanding and utilization of organoids. This discipline enhances the accuracy and efficiency of organoid imaging and contributes to significant advancements in biomedical research and clinical applications. The scope of organoid computer vision research is multi-layered and multi-staged, enhancing the study and application of organoid technology through various phases. Figure 1: Illustrates the trends and scope in organoid computer vision, highlighting key developments and areas of focus within the research community. At the first and basic stage, it involves the development and utilization of advanced imaging techniques, such as high-resolution microscopy and 3D imaging, to capture intricate visual data of organoids [ 18,19,20]. These detailed images serve as the basis for subsequent visual analysis. The next stage focuses on the development and application of sophisticated image processing algorithms to enhance organoid image quality, reduce noise, segment images, and extract visual features from the data [ 21,22,23]. This preprocessing is crucial for preparing the imaging data for comprehensive visual analysis. At a more advanced stage, computer vision analysis employs machine learning and deep learning models to automate various visual tasks. These tasks include monitoring organoid growth [ 24,25,26], counting cells, detecting and tracking cell movement, and performing quantitative analysis of organoid development and behaviour. This automation facilitates more efficient and accurate studies. The analysis involves both image and video-based methods, visual clustering, and visual comparison and retrieval against real organs for comparative studies of development. Additionally, advanced tasks such as impact analysis of drugs for visual change detection and anomaly detection for abnormal growth prediction are included. The major contributions of this paper are threefold: 1.This paper defines Organoid Computer Vision (OCV) as a new research discipline that bridges the gap between organoid research and computer vision techniques. This emphasis is crucial for highlighting the importance of OCV in enhancing the understanding of organoid biology through advanced imaging and analysis, as well as streamlining efforts to leverage developments in computer vision to push the boundaries of complex biological systems. 2.It provides a detailed taxonomy of existing work to date by categorizing studies based on types of organoids (such as brain, liver, and intestinal organoids) and computer vision techniques (including image processing, machine learning, and deep learning). This structured framework helps researchers identify trends, gaps, and opportunities in the field, offering a clear roadmap for future research and development. 3.The paper presents a comprehensive survey of significant existing work in OCV , highlighting key achieve- ments and methodologies. It also discusses current challenges faced by researchers, such as data scarcity, computational limitations, ethical and social impacts, and the need for interdisciplinary collaboration. Finally, the paper provides an outlook on future research directions, proposing innovative solutions and encouraging collaborative efforts to overcome these challenges and drive progress in the OCV discipline. By addressing these three major areas, this paper aims to establish OCV as a pivotal field that integrates organoid research with cutting-edge computer vision techniques, fostering interdisciplinary collaboration and innovation. This paper serves as the first comprehensive work to define and establish Organoid Computer Vision, providing a valuable resource for researchers and practitioners to advance the field and collaborate effectively. 2 Historical Background and Evolution of Organoids and Organoid Intelligence The field of organoid research has undergone significant evolution over the past few decades, marked by groundbreaking discoveries and technological advancements that have revolutionized our understanding of human biology and disease modelling. In 1998, a team from the University of Wisconsin, Madison, published a paper titled "Embryonic Stem Cell Lines Derived From Human Blastocysts.[ 27]" They created the first embryonic stem cells and recognized their pluripotency and capacity for self-renewal, highlighting their importance for developmental biology and drug discovery. In 2006, Japanese scientists Shinya Yamanaka and Kazutoshi Takashi published a paper [ 28] on induced pluripotent stem cells from adult mouse fibroblasts. These induced pluripotent stem cells (iPSCs) are similar to embryonic stem 3 APREPRINT - SEPTEMBER 7, 2024 Figure 2: Organoid Computer Vision Taxonomy: This taxonomy provides a detailed classification of existing research by categorizing studies based on types of organoids, such as brain, liver, and intestinal organoids, and the computer vision techniques employed, including image processing, machine learning, and deep learning. cells and can be used without the same ethical controversies. Shinya Yamanaka’s lab in Kyoto, Japan, pioneered iPS cell technology by demonstrating that introducing four specific genes encoding transcription factors (Oct3/4, Sox2, c-Myc, and Klf4) could convert adult cells into pluripotent stem cells [ 29]. This discovery earned Yamanaka and Sir John Gurdon the 2012 Nobel Prize for showing that mature cells can be reprogrammed to become pluripotent. The field of organoids began with a shift from culturing and differentiating stem cells in two-dimensional (2D) media to three-dimensional (3D) media, enabling the development of complex 3D structures resembling organs [ 30]. Stem cells can self-renew and differentiate into various cell subtypes, facilitating understanding developmental processes and disease progression. Consequently, organoids derived from stem cells allow for studying biology and physiology at the 4 APREPRINT - SEPTEMBER 7, 2024 organ level. Organoid bodies are formed by embedding stem cells in this 3D medium. Organoids enable the study of how cells interact within an organ, how they interact with their environment, how diseases affect them, and the impact of drugs. The in vitro culture of organoids makes these systems easy to manipulate and monitor. The use of 3D culture methods for structural organization became feasible with the development of extracellular matrices (ECM). The 3D medium for creating organoids can be made using an extracellular matrix hydrogel, such as Matrigel or Cultrex BME, a laminin-rich extracellular matrix secreted by the Engelbreth-Holm-Swarm tumour line. In 2006, Yaakov Nahmias and David Odde [31] demonstrated the self-assembly of a vascular liver organoid maintained in vitro for over 50 days. The next few years saw an explosion of research in the field, with scientists generating organoids that mimic a wide range of human organs, including the brain, liver, kidney, lung, and pancreas [ 2]. These organoids provided unprecedented models for studying human development, disease mechanisms, and drug responses in a controlled laboratory setting. Notable milestones include the creation of cerebral organoids, which modelled early brain development and provided insights into neurodevelopmental disorders, and the development of liver organoids capable of mimicking liver functions and disease processes. As organoid technology advanced, researchers began integrating these 3D models with bioengineering and artificial intelligence (AI) to enhance their complexity and functionality. The concept of "organoid intelligence" emerged, referring to integrating organoid systems with AI-driven computational models to simulate and analyze biological processes more accurately. This integration has facilitated organoid culture and analysis automation, enabling high- throughput screening and more precise control over experimental conditions. In recent years, organoid intelligence [ 5] has expanded to include using AI and machine learning to analyze organoid imaging data, predict developmental trajectories, and identify potential therapeutic targets. For instance, AI algorithms have been employed to analyze large-scale imaging datasets from brain organoids, uncovering neural activity and development patterns that were previously inaccessible through manual analysis. The combination of organoid models with AI has also paved the way for personalized medicine, where patient-specific organoids can be used to tailor treatments and predict individual responses to therapies. The evolution of organoid research and organoid intelligence represents a paradigm shift in biomedical science, offering new avenues for understanding human biology, modelling diseases, and developing novel therapies [ 5,1]. The continuous interplay between advancements in stem cell biology, bioengineering, and artificial intelligence promises to further enhance the capabilities of organoids, making them indispensable tools in basic research and clinical applications. Figure 2 illustrates the proposed taxonomy of organoid computer vision literature, categorizing and organizing the existing research to provide a clear framework for understanding the field. 3 Organoid Computer Vision: The Taxonomy The taxonomy for "Organoid Computer Vision" represents a structured framework designed to categorize and enhance understanding of the various aspects involved in imaging and analyzing organoids through computer vision techniques. The proposed taxonomy is divided into three primary categories, each encompassing distinct subcategories: Organotypic Specificity, Imaging Modality, and Methodology. Each of these categories provides a comprehensive view of the different dimensions involved in organoid research. Imaging Modality covers the techniques used to capture detailed images of organoids, each offering unique insights into their structure and function. Organotypic Specificity addresses the specific types of organoids under study, reflecting these models’ biological and functional diversity. Finally, Methodology encompasses the computational techniques applied to analyze these images, enabling the extraction of meaningful data and facilitating scientific discovery. This taxonomy will serve as a critical tool for organizing research efforts, identifying knowledge gaps, and advancing the application of organoid models in organoid computer vision research and therapeutic development. 3.1 Imaging Methodologies in Organoid Research This category covers the various imaging techniques used to capture and analyze organoids. Each modality provides different types of data and insights into the organoids’ structure and function. Figure 3: Illustrates representative organoid imaging using various imaging techniques, showcasing the diversity in imaging methods and the types of data they produce. Table 1 provides a summary of the unique imaging techniques for organoid imaging. In this section, we provide additional characteristics of these imaging techniques, distinctive features, uses in organoids, types of organoids where they are more effective, and their best analytical use: Multiphoton microscopy is a powerful imaging technique that enables deep tissue imaging with reduced phototoxicity and photobleaching [ 35,36,37]. This method uses multiple low-energy photons to excite fluorophores, making it 5 APREPRINT - SEPTEMBER 7, 2024 Figure 3: Representative organoid imaging from different imaging techniques. (a) Coronal views of dwi b0 and dMRI segmentation. In dMRI, this segmentation uses a standard colour-coding convention, where blue voxels reflect anisotropic areas and green/red voxels reflect isotropic areas. Here, to enhance the side-by-side comparison with the confocal imaging, the blue/green colour coding was switched. (b–j) Example images of various imaging techniques representing bright-field (panel b), wide-field fluorescence (panel c), confocal (panel d), light-sheet (panel e), OCT (panel f), FLIM (panel g), hyperspectral imaging (panel h), Brillouin (panel i), and electron microscopy (panel j). Panels b, e, and j show mouse pancreatic progenitor-derived organoids; panels c and d are hPSC-derived pancreatic spheroids. (k) Multiplex imaging in live patient-derived glioblastoma organoids. A combination of six different surface markers was stained, quenched, and imaged. This mixture of markers was divided into two separate cycles: cycle 1 is shown only (CD9, CD81, and CD133). Panels f–h show hPSC-derived retinal organoids, and panel i is an intestinal organoid. Abbreviations: FLIM, fluorescence lifetime imaging microscopy; GFP, green fluorescent protein; hPSC, human pluripotent stem cell; OCT, optical coherence tomography; RFP, red fluorescent protein. Images (a) source [ 32], (b-j) source [33], (k) source [34]. ideal for long-term live imaging of organoids. The deep tissue penetration of multiphoton microscopy is particularly useful in studying large, complex organoids such as brain or cardiac organoids, where observing dynamic processes and development over time is critical. The primary analytical use of this technique is real-time observation of developmental processes, cellular interactions, and dynamic studies within organoids. Super-resolution microscopy techniques, including STED (Stimulated Emission Depletion), STORM (Stochastic Optical Reconstruction Microscopy), and PALM (Photoactivated Localization Microscopy), break the diffraction limit of light to provide nanometer-scale resolution [ 38,39,40]. These techniques are essential for studying molecular details and interactions within organoids, allowing researchers to observe structures at the molecular level. They are most effective in organoids where detailed molecular analysis is necessary, such as in tumour or neural organoids. They are used primarily for investigating protein localization, molecular interactions, and nanostructural analysis. Light sheet microscopy offers fast, high-resolution imaging with minimal photodamage by illuminating specimens with a thin sheet of light [ 41,42,43]. This technique is particularly well-suited for long-term, real-time imaging of large organoids, such as liver or intestinal organoids, where observing developmental processes without damaging the sample is crucial. Light sheet microscopy is commonly used to study cellular interactions and structural changes over time in these organoids. Optical Coherence Tomography (OCT) is a non-invasive imaging technique that provides high-resolution, three- dimensional images, making it ideal for structural imaging of organoids [ 44,45,46]. OCT is particularly effective in organoids where internal architecture and cellular organization need to be visualized without staining, such as in kidney or retinal organoids. It is best used for internal structural analysis, allowing researchers to obtain detailed cross-sectional images and study the tissue microstructure. Fluorescence Lifetime Imaging Microscopy (FLIM) measures the decay time of fluorescence from a sample, providing contrast based on fluorescence lifetime rather than intensity [ 47,48,49]. This method is valuable for studying molecular environments within organoids, such as pH levels, ion concentrations, and protein interactions. FLIM is particularly useful in organoids where specific molecular markers are of interest, such as pancreatic or thyroid organoids. It is primarily used for analyzing molecular interactions and the local environment within the organoids. Hyperspectral imaging captures detailed spectral information across multiple wavelengths, allowing comprehensive biochemical and molecular analysis within organoids. This technique is highly effective in organoids with diverse biochemical compositions, such as tumour organoids, where understanding molecular signatures and compositions is 6 APREPRINT - SEPTEMBER 7, 2024 Table 1: Survey of Imaging Methodologies in Organoid Research, highlighting various imaging techniques. Imaging Methodology Advanced Features Uses in Organoids Limitations Multiphoton Microscopy [35, 36, 37]Deep tissue imaging, re- duced phototoxicity, and pho- tobleachingLive imaging, long-term studies, dynamic processesRequires expensive equipment and expertise Super-resolution mi- croscopy [38, 39, 40]Nanometer-scale resolution beyond the diffraction limitStudying molecular details and interactionsHigh cost, complex sample preparation, and longer imaging times Light Sheet Microscopy [41, 42, 43]Fast, high-resolution imag- ing with minimal photodam- ageLong-term, real-time imag- ing of entire organoidsLimited to relatively transparent samples, complex setup Optical Coherence To- mography (OCT) [ 44,45, 46]Non-invasive, high- resolution, three- dimensional imagingStructural imaging of organoidsLower resolution com- pared to some other tech- niques, limited to certain tissues Fluorescence Lifetime Imaging Microscopy (FLIM) [47, 48, 49]Contrast based on fluores- cence lifetimeStudy of molecular environ- ments and interactionsRequires specific fluo- rophores, complex data analysis Hyperspectral Imaging [50, 38, 51]Detailed spectral information across multiple wavelengthsAnalyzing biochemical com- positions and molecular sig- naturesHigh cost, complex data processing Third Harmonic Genera- tion (THG) Microscopy [52, 53]Label-free imaging with high contrast from intrinsic prop- ertiesDetailed imaging of cell structures and interfacesLimited to specific types of samples, relatively lower resolution Photoacoustic Imaging [54, 17]Combines optical imaging with ultrasound for high- contrast imagesVisualizing vascular struc- tures and functional imagingRequires integration of optical and ultrasound systems, complex analy- sis Confocal Microscopy [55, 56]High-resolution, optical sec- tioning, three-dimensional imagesDetailed structural study of organoidsPhotobleaching and pho- totoxicity with prolonged imaging Wide Field Fluorescence Microscopy [57, 58, 59]Fluorescence imaging for specific proteins and moleculesVisualization of specific pro- teins, nucleic acids, and moleculesLimited depth of field, photobleaching Bright Field Phase Con- trast Microscopy [ 60,61]Enhances contrast of trans- parent specimens without stainingObserving live cells and organoids, growth and mor- phology studyLimited contrast for thick or highly scattering sam- ples Atomic Force Mi- croscopy (AFM) [ 62,63]Nanometer-scale surface to- pography mappingSurface images and mechani- cal properties of organoidsLimited to surface imag- ing, slower scanning pro- cess Electron Microscopy [ 33, 64, 65]Ultra-high resolution imag- ing at the molecular levelUltrastructure analysis of organoidsRequires extensive sam- ple preparation, cannot image live samples Diffusion MRI or cycling Imaging [66, 67, 34]High-resolution, non- invasive imaging of tissue microstructureImaging internal architecture and cellular organizationHigh cost, lower resolu- tion compared to optical microscopy 7 APREPRINT - SEPTEMBER 7, 2024 crucial. Hyperspectral imaging is best used for biochemical analysis, molecular fingerprinting, and identifying unique molecular markers within the organoids [50, 38, 51]. Third Harmonic Generation (THG) microscopy is a label-free imaging technique that generates contrast from intrinsic properties of beneficial high-resolution images of cell structures and interfaces. This method is beneficial in organoids where label-free imaging is necessary, such as skin or gastric organoids, and is used primarily for detailed structural analysis of cell interfaces and organoid morphology [52, 53]. Photoacoustic imaging combines optical imaging with ultrasound, producing high-contrast images based on optical absorption properties [ 54,17]. This technique is particularly useful for visualizing vascular structures and functional imaging within organoids, such as lung or cardiac organoids. It is best used for studying vascularization, oxygenation levels, and other functional properties within these organoids. Confocal microscopy uses point illumination and a spatial pinhole to eliminate out-of-focus light, providing high- resolution, three-dimensional images. This technique is widely used for detailed structural studies of various organoids, such as kidney or lung organoids, where precise optical sectioning is necessary. Confocal microscopy is primarily used for structural analysis and optical sectioning of organoids to study their internal organization [55, 56]. Wide field fluorescence microscopy is a straightforward technique that uses fluorescence to visualize specific proteins, nucleic acids, and other molecules within organoids [ 57,58,59]. It is instrumental in organoids where specific molecular targets are being studied, such as brain or pancreatic organoids. This method is commonly used for protein and nucleic acid visualization and molecular tagging. Bright field phase contrast microscopy enhances contrast in transparent specimens without staining, making it ideal for observing live cells and organoids in real-time [ 60,61]. This technique is particularly effective in transparent and unstained organoids, such as intestinal or brain organoids, and is used primarily for observing growth, morphology, and developmental changes over time. Atomic Force Microscopy (AFM) provides nanometer-scale surface topography mapping, making it an excellent tool for analyzing surface structures and mechanical properties of organoids [ 62,63]. It is particularly useful in organoids with significant surface features like skin or epithelial organoids. AFM is best used for surface topology analysis and studying the mechanical properties of organoids. Electron microscopy offers ultra-high-resolution imaging at the molecular level, providing detailed ultrastructural analysis of organoids. This technique is particularly effective in organoids where molecular-level structural studies are needed, such as in tumour or neural organoids. It is best used for studying ultrastructure, organoid morphology, and detailed molecular analysis [33, 64, 65]. Diffusion MRI imaging is a non-invasive technique that provides high-resolution images of tissue microstructure by measuring the diffusion of water molecules. It is instrumental in large and complex organoids, such as brain or kidney organoids, where internal architecture and cellular organization must be visualized. Diffusion MRI is best used for internal structural analysis and studying cellular organization within organoids [66, 67, 34]. Overall, biosensors are increasingly used to respond to changes in the cellular environment, such as pH, calcium, or redox states. These sensors allow for real-time monitoring of cellular physiology within organoids. Similarly, the focus on live-cell imaging techniques is intensifying, with improvements in imaging systems that minimize phototoxicity and photobleaching, allowing researchers to observe organoid development and cellular processes over extended periods. Developing automated imaging systems capable of high-throughput data acquisition and analysis is crucial for screening large numbers of organoids, particularly in drug discovery and toxicology studies. Combining different imaging modalities (e.g., fluorescence, electron microscopy, and MRI) to obtain complementary information from the same organoid sample is becoming more common, providing a holistic view of organoid structure and function. The evolution of 4D imaging techniques allows researchers to visualise the 3D structure of organoids and how they change over time. This is critical for studying dynamic processes such as cell differentiation, tissue regeneration, and disease progression. 3.2 Organoid-Specificity The term "Organoid-Specificity" straightforwardly indicates that the organoid is specific to a particular organ. It can be used when discussing how certain organoids replicate the unique features of specific organs, such as liver or kidney organoids. Organoids are miniaturized and simplified versions of organs grown in vitro and mimic real organs’ complex structure and function. The specificity here refers to the particular organ or tissue type that the organoid represents. This category focuses on the types of organoids that are being studied. 8 APREPRINT - SEPTEMBER 7, 2024 Organoid-specificity is particularly important in the context of organoid computer vision and image processing because it directly influences the accuracy and reliability of data interpretation and analysis and the development of automated tools for studying organoids. Table 2 summarises various organoid types, highlighting the relevant imaging characteristics for their analysis and citing critical research studies associated with each type. Figure 4 illustrates brain organoids, visually representing their structure and the complexities of studying them. Figure 5 illustrates heart organoids, depicting their anatomy and relevance to organoid research. Figure 4: Brain organoids. (a) Fluorescence image of a cerebral organoid. Scale bar, 200µm. (b) Cortical organoid. Scale bar, 50µm. (c) MGE organoid. Scale bar, 50µm. (d) Forebrain organoid. (e) Thalamus organoid. Scale bar, 250µm. (f) Hypothalamus organoid. Scale bar, 100µm. (g) Midbrain organoid. Scale bar, 50µm. (h) Neuromuscular organoid. Scale bar, 200µm. Source [68]. Brain organoids, which include subtypes such as cortical, midbrain, hippocampal, and cerebellar organoids, are crucial for studying neurodevelopment, disease modelling, and drug testing [ 69,70,71,72]. These organoids mimic the architecture and function of the human brain, making them valuable tools for neuroscience research. Imaging techniques such as high-resolution imaging, 3D reconstruction, and segmentation are essential for analyzing the complex structures within brain organoids. Confocal microscopy and multiphoton microscopy are often used to visualize neuronal networks. Regarding computer vision, deep learning-based segmentation algorithms, like U-Net, are particularly effective in delineating different brain regions. At the same time, machine learning techniques help classify neural structures and track the development of these regions over time. Liver organoids, including hepatic, biliary, and cholangiocyte subtypes, are used extensively in drug metabolism studies, toxicity testing, and disease modelling. These organoids replicate the liver’s functional units and are valuable for understanding liver diseases and testing new drugs [ 73,15,74]. Fluorescence imaging and viability assays are commonly employed to monitor liver organoid health and function. Computer vision applications in liver organoids 9 APREPRINT - SEPTEMBER 7, 2024 Figure 5: Heart organoids. (a) Schematic illustration of a typical heart organoid shown in cross-section and front view, alongside a comparison with the early embryonic heart/foregut region (transverse plane). (b) Bright-field images (top) and fluorescent images (middle and bottom) of heart organoids. Scale bars represent 100 micrometres. Source [68]. include image classification techniques like Convolutional Neural Networks (CNNs) to differentiate between healthy and diseased tissue and segmentation algorithms to identify and quantify specific cell types. Functional analysis is also enhanced by machine learning algorithms that predict the impact of drugs on organoid viability. Intestinal organoids, comprising enteroids and colonoids, are instrumental in studying gut physiology, drug absorption, and microbiome interactions. These organoids model the complex environment of the human gut, making them essential for research in gastrointestinal diseases and drug delivery. Live-cell imaging and structural analysis are critical for observing the dynamic processes within intestinal organoids, such as cell proliferation, differentiation, and response to external stimuli [ 26,75,76]. Computer vision techniques like time-lapse imaging and 3D reconstruction, combined with advanced tracking algorithms, allow researchers to monitor cellular movements and interactions in real time, providing insights into gut health and disease progression. Kidney organoids, which include nephron, glomerular, and tubular subtypes, are valuable in kidney disease modelling, drug screening, and toxicity testing [ 77,78,79]. These organoids replicate the kidney’s filtration units, providing insights into renal function and disease mechanisms. Confocal imaging and segmentation are crucial for studying the complex structures of kidney organoids, particularly the nephron, the kidney’s functional unit. Regarding computer vision, segmentation tools like Mask R-CNN are used to accurately delineate kidney structures, while quantification algorithms measure organoid growth and function. Predictive modelling techniques are also employed to forecast disease progression and treatment response. Pancreatic organoids, consisting of endocrine and exocrine subtypes, are used in diabetes research, drug screening, and disease modelling. These organoids mimic the pancreatic tissue and are essential for studying insulin production and pancreatic diseases such as cancer. Fluorescence microscopy and functional assays are commonly used to study hormone secretion and cellular function within pancreatic organoids [ 80,81,82]. Computer vision applications include cell counting algorithms for islet cells, using tools like deep learning-based cell detection, and the application of machine learning models to predict insulin secretion levels based on morphological features captured through imaging. Lung organoids, including alveolar and bronchial subtypes, are pivotal for studying respiratory diseases, drug testing, and viral infection studies. These organoids replicate the structure and function of the human lung, making them 10 APREPRINT - SEPTEMBER 7, 2024 valuable for research in diseases like COVID-19 and other respiratory infections [ 26,83,84]. 3D imaging and functional assays are essential for visualizing the lung’s intricate structures and assessing respiratory function. In computer vision, texture analysis and object detection algorithms are used to identify and measure changes in lung tissue structure. In contrast, tracking algorithms monitor the spread of infections or the response to drugs within the organoid. Cardiac organoids, which include atrial, ventricular, and conduction system subtypes, are used in heart disease research, drug testing, and tissue regeneration studies [ 21,85,86]. These organoids mimic the heart’s electrical and mechanical properties, making them crucial for studying heart diseases and testing the cardiotoxicity of drugs. High-resolution imaging and contractility measurement are essential for analyzing the functional properties of cardiac organoids. Computer vision applications include the use of motion analysis algorithms to assess contractility and the implementation of machine learning models to predict heart organoid responses to various pharmacological treatments based on imaging data. Gastric organoids, comprising fundic (these are derived from the fundus, the upper part of the stomach) and antral (These are derived from the antrum, the lower part of the stomach) subtypes, are valuable in gastric disease modelling, drug testing, and microbiome interaction studies. These organoids replicate the stomach’s environment, making them essential for understanding gastric diseases and the effects of drugs on stomach tissue [ 87,88,89]. Live-cell imaging and structural analysis are crucial for observing the cellular architecture and interactions within gastric organoids. Computer vision techniques such as machine learning-based segmentation and classification algorithms differentiate between healthy and diseased gastric tissues, while functional assays are analyzed using image-based quantification methods. Retinal organoids, including photoreceptor and retinal pigment epithelium (RPE) subtypes, are essential for vision research, disease modeling, and drug testing. These organoids replicate the structure and function of the human retina, making them valuable for studying retinal diseases and developing treatments for vision loss [ 75,90,91]. High-resolution imaging and 3D reconstruction are critical for visualizing the complex layers of the retina within these organoids. Computer vision techniques, such as automated segmentation and object detection, are used to identify specific retinal layers and cells, aiding in the study of retinal development and the effects of therapeutic interventions. Skin organoids, consisting of epidermal and dermal subtypes, are used in skin disease modelling, drug testing, and cosmetic testing. These organoids mimic the layers of human skin, making them valuable for research in wound healing, skin cancer, and cosmetic product development [ 92,93,94]—fluorescence imaging and structural analysis study skin organoid architecture and function. Regarding computer vision, image classification algorithms are used to identify and quantify different cell types within the skin. At the same time, machine learning models predict the response of skin organoids to various treatments based on morphological features extracted from imaging data. Tumor organoids, derived from specific cancer types such as breast or colon, are critical for cancer research, drug screening, and personalized medicine. These organoids replicate the tumour microenvironment, providing insights into cancer progression and treatment response [95, 96, 75]. Multiphoton imaging and molecular analysis are essential for studying the tumour’s cellular composition and interactions within the microenvironment. Computer vision applications include deep learning-based image segmentation to delineate tumour boundaries. At the same time, machine learning models are used to predict treatment efficacy based on imaging biomarkers extracted from the organoids. Thymic organoids, including cortical and medullary subtypes, are used in immunology research and T-cell development studies. These organoids replicate the thymus environment, making them valuable for understanding immune system development and autoimmune diseases [ 97]. Confocal microscopy and functional assays are crucial for studying the structure and function of thymic organoids. Computer vision techniques, such as cell tracking algorithms, monitor T-cell migration and development within the organoids, providing insights into immune cell dynamics and interactions [98]. Thyroid organoids, comprising follicular and parafollicular subtypes, are essential for endocrine research and thyroid disease modelling. These organoids mimic the thyroid gland’s hormone-producing cells, making them valuable for studying thyroid function and disorders. Fluorescence microscopy and hormone assays are commonly used to study hormone secretion and cellular function within thyroid organoids. Computer vision applications include image segmentation algorithms to differentiate between follicular and parafollicular cells and machine learning models to predict hormone secretion levels based on morphological data from imaging. In summary, organoid specificity is vital in organoid computer vision and image processing because it ensures that the organoids closely resemble their target organs in structure and function. This fidelity allows for more accurate segmentation, classification, and quantification of imaging data, improves the training and performance of machine learning models, and enhances the reliability of disease modelling, drug screening, and personalized medicine applications. By maintaining high organoid specificity, researchers can leverage computer vision and image processing to gain deeper insights into organ development, disease mechanisms, and therapeutic responses. 11 APREPRINT - SEPTEMBER 7, 2024 Table 2: Types of organoids, their subtypes, potential applications, and the specific image analysis and computer vision techniques required for their study. Organoid TypeSubtypes / Cate- goriesPotential Applications / UsesRequired Image Analysis and Computer Vision Brain Organoids [ 69, 70, 71, 72]Cortical, Midbrain, Hippocampal, Cere- bellarNeurodevelopment studies, disease modeling, drug test- ingHigh-resolution imaging, 3D reconstruction, segmentation Liver Organoids [ 73, 15, 74]Hepatic, Biliary, CholangiocyteDrug metabolism, toxicity testing, disease modelingFluorescence imaging, via- bility assays, functional anal- ysis Intestinal Organoids [26, 75, 76]Enteroids, ColonoidsGut physiology, drug absorp- tion, microbiome interactionLive-cell imaging, structural analysis, functional assays Kidney Organoids [77, 78, 79]Nephron, Glomerular, TubularKidney disease modeling, drug screening, toxicity test- ingConfocal imaging, segmenta- tion, quantification Pancreatic Organoids [80, 81, 82]Endocrine, ExocrineDiabetes research, drug screening, disease modelingFluorescence microscopy, functional assays, islet counting Lung Organoids [ 26, 83, 84]Alveolar, BronchialRespiratory disease model- ing, drug testing, viral infec- tion studies3D imaging, functional as- says, structural analysis Cardiac Organoids [21, 85, 86]Atrial, Ventricular, Conduction systemHeart disease research, drug testing, tissue regenerationHigh-resolution imaging, contractility measurement, 3D reconstruction Gastric Organoids [87, 88, 89]Fundic, AntralGastric disease modeling, drug testing, microbiome in- teractionLive-cell imaging, structural analysis, functional assays Retinal Organoids [75, 90, 91]Photoreceptor, Reti- nal pigment epithe- liumVision research, disease mod- eling, drug testingHigh-resolution imaging, 3D reconstruction, segmentation Skin Organoids [ 92, 93, 94]Epidermal, DermalSkin disease modeling, drug testing, cosmetic testingFluorescence imaging, struc- tural analysis, viability as- says Tumor Organoids [ 95, 96, 75]Specific cancer types (e.g., breast, colon)Cancer research, drug screen- ing, personalized medicineMultiphoton imaging, molecular analysis, high- throughput screening 3.3 Computer Vision Techniques in Organoid Research Computer vision is revolutionizing organoid research by providing advanced tools for analyzing complex biological structures. Organoids, which are three-dimensional miniaturized versions of organs grown in vitro, present unique challenges in imaging due to their intricate architecture and dynamic behaviour. Integrating machine learning, deep learning, and image processing techniques into organoid research allows for more precise and automated analysis of these structures. By leveraging these technologies, researchers can enhance image quality, differentiate between various cellular components, monitor growth and development, and perform detailed phenotypic analysis [ 99]. These advancements are not only accelerating the pace of organoid research but are also paving the way for discoveries in disease modelling, drug testing, and personalized medicine. Table 3 summarizes key organoid computer vision tasks, detailing the relevant computer vision and image processing techniques and representative research studies from the literature. Figure 6 illustrates the assignment of key points 12 APREPRINT - SEPTEMBER 7, 2024 (gray lines) between five pairs of organoid images, with the bottom row displaying projections onto the first and second images, demonstrating the alignment and comparison process. By separating the imaging techniques from the computer vision techniques, we ensure that the taxonomy accurately reflects the distinct stages of data acquisition and analysis in Organoid Computer Vision. 3.3.1 Organoid Image Enhancement Organoid image enhancement [100] is a fundamental aspect of organoid research. It serves as a critical preprocessing step that dramatically improves the quality and clarity of images, ensuring that they are suitable for detailed analysis and interpretation. The complexity of organoids, with their intricate three-dimensional structures and diverse cellular compositions, necessitates using advanced image enhancement techniques to capture and understand these features fully. Techniques such as contrast enhancement and histogram equalization are employed to optimize the visibility of critical structures within the organoids, making subtle features more discernible and highlighting differences that might be indicative of specific biological processes or abnormalities. Noise reduction is another essential component of image enhancement [ 21], as raw organoid images often contain various types of noise that can obscure important details. Methods like Gaussian and median filtering are particularly effective in eliminating these unwanted artifacts, ensuring that the resulting images are clean and free from distortions that could lead to misinterpretations during analysis. By reducing noise, these techniques allow for more accurate detection of cellular structures and interactions, which is crucial for understanding the complex dynamics within organoids. Moreover, image registration and fusion techniques [ 101] play a vital role in organoid image enhancement by aligning and integrating images captured at different time points or from other imaging modalities. This alignment is crucial for tracking organoid development and changes over time, providing a comprehensive and consistent view essential for longitudinal studies. For instance, combining fluorescence microscopy images with those from phase-contrast or electron microscopy can offer a more detailed perspective on the organoid’s morphology and function. These enhancement processes directly contribute to the reliability of downstream image analysis tasks, such as segmentation and phenotypic quantification. Enhanced images allow for more precise segmentation of organoid structures, which is critical for accurately measuring features like cell size, shape, and spatial organization. Furthermore, improved phenotypic quantification facilitates a better understanding of the organoid’s functional properties, enabling researchers to draw more robust conclusions about its biological behaviour [102]. In the context of organoid research, where accurate modelling of human tissues and organs is paramount, the role of image enhancement cannot be overstated. By refining the quality of organoid images, these techniques empower researchers to conduct more thorough and precise analyses, leading to deeper insights into organ development, disease mechanisms, and potential therapeutic strategies. As organoid research continues to evolve, advancements in image enhancement techniques will remain essential for pushing the boundaries of what can be observed and understood from these complex biological models. 3.3.2 Organoid Image Differentiation Organoid image differentiation [ 91,116] is a critical process in organoid research, focusing on the precise identification and distinction of various structures within an organoid. This process leverages advanced imaging techniques such as digital staining [ 117], labelling [ 115], and data augmentation [ 120] to enhance the ability to identify and analyze specific cellular components, even in situations where traditional physical staining methods are not feasible. Digital staining, for example, allows researchers to apply virtual dyes to images, highlighting different cell types or substructures without the need for actual chemical stains. This is particularly useful in preserving the integrity of live organoids or when working with limited sample quantities. Labelling techniques [ 115], often combined with fluorescent markers, further enhance the differentiation of cellular components by tagging specific proteins or structures within the organoid. These labels make it possible to track particular cell types’ behaviour or observe dynamic processes such as cell division, migration, or differentiation in real- time. The use of data augmentation techniques, such as rotation, scaling, and flipping of images, expands the diversity of the dataset, which is essential for training robust machine learning models capable of accurately distinguishing between different organoid structures. One of the most advanced methods employed in organoid image differentiation is the use of Generative Adversarial Networks (GANs) [ 120,22] for image synthesis. GANs are powerful tools that can generate realistic synthetic images based on existing data, effectively increasing the volume and variability of training data for machine learning applications. This is particularly beneficial in scenarios where obtaining large quantities of labelled images is challenging. 13 APREPRINT - SEPTEMBER 7, 2024 Table 3: Survey of Methodologies in Organoid Computer Vision, highlighting key tasks and corresponding techniques for image enhancement, differentiation, phenotyping, growth monitoring, and additional advanced techniques. Organoid Computer Vi- sion TaskComputer Vision Methodologies Representative Work Organoid Image En- hancementImage enhancement (contrast enhancement, histogram equalization); Noise reduction (Gaussian filtering, median filtering, Non- Local Means filtering); Image normaliza- tion; Image registration (rigid, non-rigid); Image fusion; Multiview image fusion[100,103,104,105,21, 95,106,107,103,108, 101,109,15,110,111, 112, 113, 114] Organoid Image Differ- entiationDigital staining; Labeling and annotation; Data augmentation (rotations, flips); feature extraction, Image synthesis (GANs);image cytometry 3D reconstruction; Image-to- image translation; Multiview reconstruction[115, 116, 117, 118, 119, 120,121,122,123,124, 125,76,126,127,22,75, 95, 120, 128, 129] Organoid PhenotypingOrganoid segmentation (U-Net, Mask R- CNN); phenotypic quantification, Seman- tic segmentation; Classification; Detection; Single cell localization; Multiview analysis[130, 102, 131, 132, 133, 134,135,136,137,138, 25,139,140,141,142, 143,144,145,146,147, 148, 149, 150, 102] Organoid Growth Mon- itoringBoundary detection; Cell and organoid tracking (Kalman filters, optical flow); Anomaly detection; Quantitative analysis (cell count, size, growth rate); Temporal analysis (time-lapse imaging, motion analy- sis); Multiview temporal analysis[44,151,152,153,154, 26,153,155,156,157, 158, 159, 72, 160, 161] Additional TechniquesMultimodal Imaging Analysis (Data fusion, Correlation analysis); Automated Workflow Integration (High-throughput analysis, Ro- bust data management); Deep Learning- Based Analysis (Advanced segmentation models, Image dataset development)[162,163,164,165,80, 166,167,168,169,170, 171, 172] By generating new, high-quality images, GANs help improve the robustness and model of machine learning models used in organoid analysis. Feature extraction [ 173] and multiscale reconstruction techniques [ 128] are also pivotal in organoid image differentiation. Feature extraction involves identifying and quantifying key attributes of cells and tissues, such as texture, shape, and intensity, which can then be used to classify and differentiate between various structures within the organoid. Multiview reconstruction, on the other hand, allows for creating detailed 3D models of organoids by integrating images taken from different angles or perspectives. This comprehensive 3D modeling [ 76] provides a deeper understanding of the organoid’s internal architecture, enabling researchers to study the spatial relationships and interactions between different cell types and structures. Figure 7 illustrates a cross-laboratory brain organoid imaging dataset that demonstrates the robustness of organoid analysis pipelines across diverse phenotypes and imaging features derived from a newly developed dataset. These differentiation techniques are indispensable for accurately classifying and studying organoids’ diverse cell types and structures. By distinguishing between various components of an organoid, researchers can gain insights into the complex biological processes that govern organ development, function, and disease. Moreover, accurate differentiation is essential for applications such as drug testing, where understanding the specific effects of treatment on different cell types within an organoid can lead to more targeted and effective therapies. 14 APREPRINT - SEPTEMBER 7, 2024 Figure 6: Shown above, from left to right, are the assignments (gray lines) between key points in five pairs of organoid images. The bottom row displays projections onto the first and second images. Columns 1-4 illustrate assignments between morphologically similar organoids, while the last column shows an assignment between dissimilar organoids, indicated by larger distances between assigned key points. For clarity, images are rotated, and only key points of cell nuclei are displayed. Source [80]. 3.3.3 Organoid Phenotyping Organoid phenotyping [ 50,69] is a critical component of organoid research, focusing on the detailed analysis of organoids’ morphology and function. This process is essential for understanding how organoids develop, function, and respond to various stimuli, including drugs or disease conditions. The success of organoid phenotyping relies heavily on advanced segmentation, classification, and detection algorithms, which enable researchers to isolate and study specific regions or cells within the complex three-dimensional structure of an organoid [ 45]. Figure 8 illustrates a qualitative comparison that evaluates detection and cell counting models specifically adapted for the BOrg dataset, highlighting the performance of these models. Segmentation techniques, such as U-Net [ 130,?] and Mask R-CNN [ 79], YOLO [ 144] play a crucial role in organoid phenotyping by providing precise and accurate delineation of organoid boundaries and internal structures. U-Net, a convolutional neural network designed explicitly for biomedical image segmentation, excels in capturing the fine details of organoid morphology, making it possible to isolate distinct regions or cell types for detailed analysis. Mask R-CNN, another powerful tool, extends this capability by segmenting individual objects within an image and identifying and classifying them, thereby allowing for a more comprehensive understanding of the organoid’s cellular composition. Figure 9 illustrates automated organoid identification and analysis, showing the process and techniques used for identifying and analyzing organoids within datasets. Phenotypic quantification [ 24] and semantic segmentation [ 153]are integral to organoid phenotyping, as they provide deep insights into the structural and functional characteristics of organoids. Through phenotypic quantification, researchers can measure and analyze specific features of organoids, such as size, shape, and cell density, which are crucial for understanding how these miniaturized organs replicate the behaviour of their natural counterparts. Semantic segmentation goes a step further by categorizing each pixel in an image into meaningful classes, such as different cell types or tissue regions, which is particularly useful in identifying patterns and anomalies in disease models or evaluating the effects of experimental treatments. Figure 10 illustrates actual nucleus DAPI staining, using a fluorescent dye to bind DNA and highlight cell nuclei, serving as a reference for comparing virtual nucleus painting methods. Moreover, using single-cell localization [ 174,24] and image analysis [ 15]significantly enhances understanding of cellular behaviour within organoids. Single-cell localization techniques allow for precisely identifying and tracking individual cells within an organoid, providing insights into how cells move, interact, and differentiate over time. This 15 APREPRINT - SEPTEMBER 7, 2024 Figure 7: The cross-laboratory brain organoid imaging dataset illustrates organoid analysis pipelines’ robustness across heterogeneous phenotypes and imaging features. The images focus on Organoid 3 at day 30, with the first column displaying results from both imaging labs. Columns 2 through 6 present various segmentation methods: Column 2 shows the Ground Truth (GT) organoid segmentation, Column 3 depicts CellProfiler predictions, Column 4 features OrganoSeg predictions, Column 5 illustrates MOrgAnaMLP,W predictions, and Column 6 displays SegFormer predictions. On day 2, the dark textured region surrounding the central circular area indicates cell debris around the embryoid body. By day 30, the translucent circumferential structures represent the Matrigel matrix in which the organoids are embedded. The scale bar represents 500 micrometers. Source: [172]. Figure 8: The qualitative comparison evaluates detection and cell counting models adapted for the BOrg dataset. This analysis highlights the performance of different models in accurately detecting and counting cells within the organoid images. Please refer to source [171] for detailed comparisons and results. level of detail is crucial for studying complex processes such as tissue development, immune responses [ 60], and cancer progression. Multiview analysis [ 175], which involves capturing and integrating images of organoids from multiple angles or time points, further enriches the phenotypic data by offering a more complete and dynamic view of organoid architecture and cellular dynamics. These phenotyping techniques [ 69,88,168]are essential for advancing our understanding of organoids and their potential applications in biomedical research. These methods provide the foundation for studying disease mechanisms, testing new drugs, and developing personalized medicine approaches by enabling detailed and accurate analysis of organoid structure and function. For instance, in drug discovery, precise phenotyping allows researchers to observe how different compounds affect organoid development and function, leading to better drug efficacy and safety predictions. 3.3.4 Organoid Growth Monitoring Monitoring the growth and development of organoids [ 44,155,156]is a dynamic and intricate process that involves tracking changes in their structure and behaviour over time. This essential aspect of organoid research allows scientists to observe how these complex models develop, respond to various stimuli, and replicate the growth patterns of real organs or tissues. The process relies heavily on advanced computer vision techniques, such as boundary detection, cell tracking, and anomaly detection, which are crucial for accurately capturing and analyzing the ongoing changes within organoids. 16 APREPRINT - SEPTEMBER 7, 2024 Boundary detection [ 131,176] plays a fundamental role in organoid growth monitoring by enabling researchers to delineate the edges of organoids and their internal structures with precision. This is critical for assessing how the organoid expands, contracts, or changes shape as it matures. Cell tracking further enhances this process by following the movement and division of individual cells within the organoid, providing insights into cellular dynamics and how they contribute to the overall architecture of the organoid. Quantitative analysis tools [ 67,71]are indispensable for measuring key growth metrics, such as growth rates, cell counts, and cell sizes. These tools provide objective data that allow researchers to quantify the pace and extent of organoid development, compare different experimental conditions, and assess the impact of various treatments or genetic modifications. Temporal analysis, including time-lapse imaging and motion analysis, is crucial in providing a detailed view of organoid development. Time-lapse imaging captures the continuous evolution of organoids over time, allowing researchers to observe subtle changes that occur as the organoid grows. Motion and change analysis [177,178] further enriches this data by examining the movement patterns within the organoid, shedding light on the dynamic processes that drive growth and differentiation. Figure 9: Automated Organoid Identification and Analysis: (a) OrganoID can identify individual organoid contours, even when in contact. The left image demonstrates the steps of identification, showing neural network predictions (second from left) with less confidence at boundaries (white arrows), facilitating edge detection with a Canny filter (second from right). This leads to a final single-organoid labelled image (right) using a watershed transformation. (b) Compared to manual counts, OrganoID was used to count organoids in test images. The concordance correlation coefficient (CCC) and Bland-Altman analysis (right) show measurement agreement and low bias. (c) Organoid area measurements by OrganoID and manual methods were compared, showing CCC computation (left) and Bland-Altman analysis (right). (d) Time-lapse microscopy images track individual organoids over time, shown at three time points. (e) Growth curves for selected organoids from (d) were automatically measured. source citematthews2022organoid. Anomaly detection algorithms [ 179] are particularly valuable in organoid growth monitoring, as they help identify deviations from standard growth patterns. These deviations could indicate potential issues such as disease onset, the 17 APREPRINT - SEPTEMBER 7, 2024 effects of a specific treatment, or structural abnormalities [ 180,181]within the organoid. Early detection of anomalies allows researchers to investigate the underlying causes and make informed decisions about proceeding with their experiments. This capability is critical in drug testing, where identifying adverse effects on organoid growth can provide vital insights into the safety and efficacy of new therapies [182, 183]. 3.3.5 Advanced Analytical Techniques and Multimodal Integration Advanced computer vision techniques, including multimodal imaging analysis [ 168,37] and deep learning-based models [ 71,117,131], are pivotal in pushing the boundaries of organoid research. Multimodal imaging combines data from various modalities, providing a richer and more comprehensive view of organoids by integrating and correlating information from different imaging techniques. This holistic approach enhances the understanding of organoid structure and function. Automated workflow integration and high-throughput analysis streamline the processing of large datasets, making research more efficient and scalable. In addition to these methods, advanced concepts like zero-shot learning [ 163,184,185], domain adaptation [ 186] and meta-learning [ 187], are increasingly being applied in organoid research. Zero-shot learning enables models to accurately classify new, unseen data based on prior knowledge, particularly useful when dealing with novel organoid types or conditions. Domain adaptation and domain generalization techniques allow models to perform well across different imaging conditions or datasets, enhancing their robustness and applicability. Meta-learning further optimizes these models by enabling them to adapt to new tasks or datasets with minimal retraining quickly. These advanced analytical techniques and learning approaches are revolutionizing organoid research, enabling more accurate, automated, and scalable analysis. Figure 10: The real nucleus DAPI staining (a), which uses a fluorescent dye to bind DNA and highlight cell nuclei, serves as the ground truth. PhaseFIT (b) more accurately captures the crypt-villus structure than GAN methods (c and d). The white dashed lines indicate regions where the virtual painting successfully detected the crypt areas, while the yellow dashed lines show where the crypt regions were missed. Source [22] 4 Challenges in Organoid Computer Vision Organoid computer vision is a rapidly evolving field that combines the complexities of biological systems with advanced computational techniques to analyze and interpret the intricate structures of organoids. These three-dimensional cell 18 APREPRINT - SEPTEMBER 7, 2024 cultures, which mimic the architecture and function of real organs, have the potential to revolutionize biomedical research, offering new insights into development, disease modelling, and drug discovery. However, integrating computer vision into organoid research presents a unique set of challenges that must be addressed to fully realise this potential [7,188]. These challenges span from ensuring data quality and managing the inherent variability of biological samples to overcoming the computational complexity of processing vast amounts of imaging data [ 170]. Additionally, there is a critical need to integrate biological insights with computational analysis to ensure the results are both interpretable and actionable by researchers. As the field advances, addressing these challenges through innovative solutions will be essential for unlocking the full capabilities of organoid computer vision. 4.1 Data Quality and Variability One of the primary challenges in organoid computer vision is maintaining high data quality while managing the inherent variability in organoid cultures [ 141]. These cultures often exhibit significant heterogeneity due to differences in cell types, growth conditions, and experimental protocols. Such variability can lead to inconsistencies in imaging data, complicating efforts to standardize analysis and derive reliable conclusions. High-quality imaging is essential for accurate processing and analysis, yet achieving consistent quality across different samples and experiments remains a significant hurdle. To address this, researchers must develop robust techniques capable of handling and correcting for variability, ensuring that the data used for analysis is high-quality and representative of the biological phenomena under study. 4.2 Computational Complexity and Resource Requirements The computational demands of analyzing organoid imaging data present another significant challenge. As complex three- dimensional structures, organoids generate vast amounts of high-resolution imaging data [ 189,190]. Processing and analyzing this data requires substantial computational resources, including advanced hardware and efficient algorithms. While machine learning and deep learning models are practical tools for such tasks, they are computationally intensive and often necessitate specialized hardware, such as GPUs. Additionally, managing and storing the large datasets organoid research produces requires significant storage capacity. Balancing the need for detailed, high-resolution analysis with the available computational resources is a key challenge that must be addressed to advance the field of organoid computer vision. 4.3 Integration of Biological Insights and Interpretability Integrating biological insights with computational analysis and ensuring the interpretability of results are critical challenges in organoid computer vision [ 75]. Computational models and algorithms must not only process and analyze data with precision but also generate biologically meaningful insights. This necessitates a deep understanding of the studied biological processes and the computational techniques employed. Furthermore, the results produced by these models must be interpretable by biologists and medical researchers to effectively guide further experiments and clinical applications. Achieving this integration involves developing models that can incorporate biological knowledge and present results in an understandable and actionable manner, thereby bridging the gap between computational analysis and biological interpretation. 5 Applications of Organoid Computer Vision Organoid computer vision is advancing biomedical research by enabling precise and detailed analysis of the complex three-dimensional structures of organoids [ 12,17,86]. As these miniaturized models of human organs become increasingly central to studies in development, disease, and therapeutic testing, applying sophisticated computer vision techniques is essential. Tools such as image segmentation, classification, and cell tracking allow researchers to delve deeper into the nuances of organoid behaviour, extracting critical insights with a level of precision and efficiency that was previously unachievable. These advancements are transforming many applications—from high-throughput drug screening and personalized medicine to the intricate study of disease mechanisms and developmental biology. By automating and enhancing organoid analysis, computer vision accelerates research and expands the boundaries of what can be explored and understood in human biology, making it a cornerstone of modern biomedical science. 5.1 Organoid Morphological Analysis Organoid Computer Vision plays a crucial role in the morphological analysis of organoids. This application involves using advanced imaging techniques and computational algorithms to study organoids’ structure, shape, and organization. 19 APREPRINT - SEPTEMBER 7, 2024 By analyzing morphological features, researchers can gain insights into the development and differentiation of organoids, identify abnormalities, and understand the effects of various treatments [ 167,75]. High-resolution imaging and sophisticated image processing allow for detailed visualization and quantification of morphological changes over time, providing a comprehensive understanding of organoid biology. 5.2 Disease Modeling and Drug Screening Organoids serve as powerful models for studying human diseases, and Organoid Computer Vision enhances this capability by enabling precise and automated analysis of disease phenotypes. Researchers can use organoids to model various diseases, including cancer, neurodegenerative disorders, and infectious diseases [ 11,48,159]. Organoid Computer Vision facilitates high-throughput drug screening by automating the detection of disease markers and the evaluation of drug efficacy. By analyzing the effects of different compounds on organoids, researchers can identify potential therapeutic candidates and optimize treatment regimens, accelerating the drug discovery process. 5.3 Personalized Medicine and Therapeutics One of the most promising applications of Organoid Computer Vision is in personalized medicine and therapeutics. By creating patient-specific organoids from stem cells, researchers can model individual patients’ treatment responses and develop novel applications in regenerative medicine[ 30,5]. Organoid Computer Vision enables detailed analysis of these responses, allowing for the identification of personalized therapeutic strategies. This approach can help predict how patients respond to drugs, minimize adverse effects, and tailor treatments to individual needs. The integration of computational analysis with patient-derived organoids has the potential to revolutionize personalized medicine, leading to more effective and targeted therapies. 5.4 Integration with Other Omics Data Organoid Computer Vision can be integrated with other omics data, such as genomics, transcriptomics, proteomics, and metabolomics, to provide a comprehensive understanding of organoid biology [191, 192]. This multi-omics approach allows researchers to correlate morphological and functional data with molecular profiles, uncovering the underlying mechanisms driving organoid development and disease. By combining imaging data with omics data, researchers can identify biomarkers, elucidate signalling pathways, and gain insights into the complex interactions within organoids. This holistic view enhances our understanding of organoid systems and facilitates the development of novel therapeutic interventions [193]. 6 Outlook and Future Directions in Organoid Computer Vision The future of organoid computer vision is poised for significant advancements, particularly by developing more sophisticated models that can more accurately mimic the complexity of human tissues and organs [ 18,194]. As stem cell technology, bioengineering, and computational modelling continue to evolve, these advanced models will provide increasingly precise representations of human biology. This will enable deeper insights into developmental processes and disease mechanisms, offering researchers more reliable tools for studying complex biological systems like the human brain [195]. A key area of future research will focus on integrating multi-scale data, combining high-resolution imaging with molecular, genetic, and temporal information to capture the dynamic nature of organoid systems [ 196]. This approach will allow for the development of predictive models that simulate organoid behaviour under various conditions, providing valuable insights for basic research and clinical applications. Such integration will enhance our ability to understand and manipulate organoids, leading to more comprehensive and predictive models of human health and disease [11]. To overcome these challenges, emerging techniques such as zero-shot learning, domain adaptation, meta-learning, and domain generalization are increasingly being applied in organoid computer vision [ 185,186]. Zero-shot learning [197] enables models to classify new, unseen data based on existing knowledge, which is particularly valuable in dealing with the novel and variable nature of organoids. Domain adaptation and domain generalization [ 198] enhance model robustness, allowing them to perform well across diverse imaging conditions and datasets. Meta-learning optimizes these models to adapt to new tasks or datasets with minimal retraining quickly, improving their efficiency and applicability. Together, these advanced analytical techniques offer promising solutions to data quality, computational complexity, and the integration of biological insights, driving forward the capabilities and impact of organoid computer vision. 20 APREPRINT - SEPTEMBER 7, 2024 Figure 11: Timeline of In-Silico Organoids: Simulation Results at Days 3, 5, and 7. In-silico organoids are computational models that simulate the development and behaviour of organoid structures in a virtual environment. This timeline compares the simulation outcomes of the ’original model’ (a-c) and the ’proposed model’ (d-f) at three different time points: day 3 (a,d), day 5 (b,e), and day 7 (c,f). The colour coding indicates different cell types and structures within the simulated organoid: Stem cells (blue), transit amplifying cells (yellow), Paneth cells (green), general differentiated enteroid cells (purple), Matrigel (pink), lumen (dark blue), and simulation boundary (red). Source [76] Advancements in real-time monitoring [ 22] and control of organoid cultures will also be crucial in the coming years. The development of automated systems capable of continuous imaging and analysis will allow researchers to monitor organoid development in real time, making necessary adjustments to optimize conditions as they evolve. Machine learning algorithms will play a significant role in these real-time control systems, improving reproducibility, efficiency, and the overall outcomes of organoid research [39]. In-silico organoids [ 76] are virtual models that simulate the growth, development, and behaviour of organoids using computational techniques organs on a chip [ 199] to a human on a chip [ 200]. Figure 11 illustrates a timeline of in-silico organoids, displaying simulation results and the evolution of computational models used in organoid research. These models are becoming increasingly important in biomedical research due to their ability to replicate complex biological processes without physical samples. In-silico organoids are particularly valuable for studying organ development, disease progression, and drug responses in a controlled and reproducible environment. Their development is closely related to image analysis, as the accuracy of in-silico models often relies on high-quality imaging data from real organoids. Researchers can refine these virtual models by analyzing images of actual organoids to mimic biological realities better. Furthermore, in-silico organoids generate their own imaging data sets, which can be analyzed to predict how real organoids might behave under various conditions. This synergy between in-silico organoids and image analysis allows for more precise and scalable research, facilitating advancements in personalized medicine and regenerative therapies. Organoid computer vision is expected to have a profound impact on regenerative medicine. By leveraging organoids’ ability to mimic native tissue architecture and function, researchers will explore their use in tissue repair and replacement. Integrating advanced computational analysis with organoid models will facilitate the design and optimization of regenerative therapies tailored to individual patients, paving the way for personalized regenerative medicine. This could revolutionize how we approach tissue engineering and the treatment of degenerative diseases. 21 APREPRINT - SEPTEMBER 7, 2024 As the field progresses, addressing ethical and regulatory considerations will become increasingly important [ 201]. Creating and manipulating complex organoid systems, particularly those that model sensitive human tissues such as the brain or reproductive organs, will require careful ethical deliberation. Developing standardized guidelines and regulatory frameworks will be essential to ensure the responsible use of organoid technology in both research and clinical settings. This will involve balancing the potential benefits of these technologies with the ethical implications of their use, ensuring that advances in organoid computer vision are made with societal and ethical considerations in mind [202, 203, 201]. Brain organoid research holds significant potential for advancing our understanding of artificial intelligence (AI) and possibly even artificial consciousness. Although true artificial consciousness remains a theoretical concept, brain organoids could provide a pathway toward understanding the neural correlates of consciousness. Brain organoids can be a biological blueprint for developing neuromorphic computing systems that mimic the brain’s architecture and function in hardware [204]. Overall, the outlook for organoid computer vision is highly promising, with significant potential to transform biomedical research and regenerative medicine. By continuing to develop sophisticated models, integrate multi-scale data, and enhance real-time monitoring and control, the field will push the boundaries of what is possible in understanding and manipulating complex biological systems. Addressing the ethical and regulatory challenges will be essential to harnessing the full potential of these technologies responsibly and beneficially. 7 Discussion and Concluding Remarks In this survey, we have explored the emerging field of Organoid Computer Vision, which integrates advanced com- putational techniques with organoid biology to analyze and interpret organoid imaging data. The development of Organoid Computer Vision has been driven by significant milestones in stem cell research, bioengineering, and artificial intelligence, which enable the creation and detailed study of complex 3D organoid structures. We have presented comprehensive taxonomies categorizing the field based on the types of organoids studied and the computer vision techniques employed. These taxonomies provide a structured framework for understanding the diverse applications and methodologies in Organoid Computer Vision, highlighting the integration of high-resolution imaging, image processing, and machine learning models. Despite the promising advancements, several challenges remain, including ensuring data quality and managing variability, addressing the computational complexity and resource requirements, and effectively integrating biological insights with computational analysis to enhance interpretability. Addressing these challenges will be crucial for advancing the field and achieving its full potential. Looking forward, developing more sophisticated organoid models, integrating multi-scale data, advancements in real- time monitoring and control, applications in regenerative medicine, and addressing ethical and regulatory considerations will drive the future directions of Organoid Computer Vision. These advancements will enhance our understanding of human biology and disease mechanisms and pave the way for personalized medicine and innovative therapeutic approaches. This survey aims to provide a valuable resource for researchers and practitioners, fostering interdisciplinary collaboration and innovation in Organoid Computer Vision. By establishing this field as a distinct discipline, we hope to encourage further exploration and development, ultimately leading to significant scientific and medical breakthroughs.