Closed-loop optogenetic control of cell biology enables outcome-driven microscopy

Domain: physics
Published: 2025-12-23
Editorial outcome: major_revision

Abstract
Smart microscopy is transforming biological imaging by integrating real-time analysis with adaptive acquisition to enhance imaging efficiency. Whereas many emerging implementations are event-driven and focus on on-demand data acquisition to reduce phototoxicity, we here present ‘outcome-driven’ microscopy, a framework combining smart microscopy with optogenetics to control cell biological processes and achieve predefined outcomes. We validate this approach using light-based control of cell migration and nucleocytoplasmic transport, demonstrating robust spatiotemporal control of cellular behaviour in single cells and in cell populations.

Introduction
Biological imaging is constantly evolving, with “smart” microscopes paving the way for efficient, adaptive workflows. A smart microscopy platform performs real-time analysis, adjusting acquisition parameters on the fly1–6. Typical implementations of such real-time feedback loops use the information from the acquisition to change microscope behaviour, such as imaging modalities, speed, or dimensions. This allows the microscope to adapt to the sample and minimise imaging that does not contribute to answering the biological question, thus preserving sample health by reducing phototoxicity and improving efficiency1,5,7,8. As such, the majority of smart microscopy implementations record the sample passively, for example, to track objects9–13, continually optimise imaging parameters10,14–17 or, in the case of event-driven microscopy, adjust modalities in response to specific events18–20.

Beyond observation of a biological process, microscopy can also facilitate active control. Most notably, optogenetics provides a platform for inducible control of biological systems across scales21–25 using light, with a dose-response relationship26–29 and a rapid localised response of cells to spatial light patterns30–33. Optogenetics has been previously combined with closed-loop smart microscopy platforms in a passive manner, with the controller adjusting hardware on the fly through automated segmentation or tracking34–38. However, the control of a biological system available through optogenetics provides the opportunity to extend feedback control to optimise not only hardware in passive observation, but also actively control biological processes themselves to achieve user-defined outcomes22,39. Indeed, combinations of closed-loop control and optogenetics have previously enabled such “outcome-driven” research, adjusting light intensity or patterns on the fly in vivo40–47, or in cultured cells48–63 as demonstrated in cybergenetics64—most often with specific custom platforms designed for intensity optimisation at the population or whole-cell level for gene expression. As such, none of these examples fully harness the potential of optogenetics to achieve automated subcellular control of cellular dynamics. Such control over biology has been a long-term goal of cell biological research65.

Here, we present a generalised platform for outcome-driven microscopy that uses optogenetics and real-time feedback to achieve automated spatiotemporal control of subcellular cell biology. This platform can control both the spatial patterning and intensity of light and adjusts them on the fly to bring biological systems to predefined outcomes, as demonstrated by long-term guidance of cell migration over predefined paths and by the controlled titration of protein levels in the cytosol or nucleus.

Results
A flexible smart microscopy platform for outcome-driven experiments
First, we built a modular smart microscopy platform, with interchangeable modules for microscope communication strategies, image processing pipelines, and control approaches (Supplementary Fig. 1 and Supplementary Note 1). This platform can be adapted for a variety of smart microscopy experiments beyond our implementation by adjusting the modules to fit specific cases. For our implementation of outcome-driven microscopy, a feedback loop is established whereby the microscope captures an image for processing, and based on the information extracted from the image, the control module optimises specific biological processes to achieve predefined outcomes by adjusting the illumination parameters of a spatial light modulator.

Directing single-cell migration with real-time feedback and optogenetic control
To test the capabilities of outcome-driven microscopy, we used cell migration as our first proof-of-concept. Control of cell polarisation and migration using optogenetics is well established through the use of photoactivatable RhoGTPases or recruitment of upstream effectors to the plasma membrane66–72 (Fig. 1a). Light directed migration has been achieved by gradient illumination70, manually updating the area of illumination (AOI)67–69,73,74, or with smart microscopy, using live segmentation with tracking to achieve constant illumination in the same cellular area34–36. We reasoned that outcome-driven microscopy could push this further, and constantly update the AOI to direct cells to predefined migration paths. This approach would allow us to overcome stochastic variations such as differences in protein expression levels, morphology, or response times, with the controller ensuring that cells stay on the predefined track. To establish this, we used feedback on the position of the cell centroid relative to a path made up of setpoints, combined with a trajectory tracking controller to selectively illuminate the cell in the region closest to the next selected setpoint (Fig. 1b and Supplementary Note 1). For reliable segmentation, we used the pre-trained AI segmentation method Segment Anything Model (SAM)75, with a custom extension of its interface for tracking (Supplementary Note 2). Using the migratory cell line HT1080 stably expressing an optogenetic construct to recruit the RAC1 effector TIAM1 to the plasma membrane72, we found that we could reliably guide cells around a specific path multiple times for >10 h (Fig. 1c and Supplementary Movie 1), maintaining a relatively consistent speed (Fig. 1d and Supplementary Fig. 2a). Quantifying the distance between the centroid and the desired path (path deviation) revealed that the centroid of the cell was kept within 2.5 µm of the path (Supplementary Fig. 2b), demonstrating precise control of directed cell migration.

Robust feedback control across heterogeneous cell populations
Outcome-driven microscopy provides the opportunity to control multiple cells in the same manner to the same setpoint, while also considering cell-cell variation. Indeed, when controlling multiple cells with the same constant LED irradiance (0.23 µW/cm2), we observed some variation in cell speed, independent of the expression level of TIAM1 (Supplementary Fig. 2c, d). Despite this heterogeneity, the controller was able to keep all cells within 5 µm of the desired path throughout the experiment (Fig. 1e). Path deviation was consistently low between cells, even at the lowest irradiance (Fig. 1f and Supplementary Fig. 2e), indicating that even a small degree of optogenetic activation is sufficient to maintain a consistent signalling pathway and guide cells to specific paths with outcome-driven microscopy.

Adaptive modulation of migration speed via optogenetic feedback
Next, we asked if, despite cell-cell variation in speed when keeping irradiance constant, we could modulate cell speed by adjusting irradiance over time, due to the dose-response relationship of optogenetics. To evaluate this relationship, we guided cells around a circular path, and adjusted irradiance in a stepwise manner, changing the value for each complete or half loop (Fig. 1g and Supplementary Movie 2). Indeed, a greater variation in speed is seen when guiding cells with different light levels (Supplementary Fig. 2f), and notably, we observed some degree of irradiance-dependence of relative cell speed (Fig. 1g, h), demonstrating that adjusting irradiance can speed up or slow down cells. Interestingly, the inherent variability in the baseline speed of unstimulated, undirected cells (Supplementary Fig. 2g) prevented us from consistently achieving specific absolute speed setpoints at fixed irradiance levels (Fig. 1i). We observed that cells that are initially faster are more capable of decreasing their speed, and cells that initially move slow are easier to speed up, indicating a saturation point. Overall, the non-linear relationship we have observed between irradiance and control of speed may reflect underlying mechanisms or regulatory processes that buffer against external modulation. Variability in factors such as expression levels of regulators, polarity components, or metabolic state could influence how each cell interprets or responds to optogenetic input. Additionally, feedback within migration signalling networks may limit the extent to which light-induced perturbations can push cell behaviour beyond certain physiological constraints. As prolonged blue light exposure and sustained TIAM activation could also contribute to the heterogeneity that we observed by perturbing cell behaviour or inducing cell death, we evaluated phototoxicity by exposing HT1080-TIAM cells to whole-field illumination at low irradiance for 24 h. A minimal increase in toxicity was detected compared to non-exposed controls on the same coverslip, indicating that phototoxicity is not responsible for the observed heterogeneity and supporting the feasibility of long-term experiments (Supplementary Fig. 2h–j).

Parallel control of multiple migrating cells with real-time coordination
To demonstrate the high level of control that we can achieve and to push the limits of outcome-driven control of cell migration, we adapted our platform for outcome-driven microscopy to control multiple cells simultaneously, each on a different path. Additionally, to avoid collisions we added coordination between cells by incorporating a lookahead, whereby the cells will be pulled back in the direction opposing their initial movement if an imminent collision is detected (Fig. 2a). We observed that we could effectively guide multiple cells to their respective paths with minimal error, and effectively stall them to avoid collisions (Fig. 2b and Supplementary Movie 3).

Furthermore, we considered that a future application of this approach for investigating cell dynamics is to explore cell-cell interactions during migration. To test this, we controlled cells to directed paths in a crowded environment, in the presence of cells that are not expressing the optogenetic construct and observed cell interactions during directed migration that influenced both the migration of uncontrolled cells and the morphology of the controlled cell (Fig. 2c and Supplementary Movie 4). MCF7, an epithelial cell type that displays collective cell migration76 would represent an interesting model for studying these interactions. As a preliminary step, we demonstrated that our method can successfully direct migration in single MCF7 cells (Supplementary Movie 5), establishing a foundation for future work investigating collective behaviour.

Dynamic feedback control of protein localisation using optogenetic export systems
In addition to outcome-driven microscopy aimed at controlling whole-cell dynamics with spatial illumination patterns, we next set out to establish a subcellular proof-of-concept and explore an intensity-driven relationship in depth to further demonstrate flexibility in control strategy. The blue light-induced nuclear export system (LEXY)77 has a rapid light-response, and induces striking subcellular localisation differences with a dose-response relationship (Fig. 3a). We hypothesised that we could achieve full control of protein levels within the cytosol and nucleus using outcome-driven microscopy by adjusting illumination irradiance on the fly, placing irradiance under control of a proportional-integral-derivative (PID) controller78 (Fig. 3b). The controller is optimised to minimise the error of the intensity in either the nucleus or cytosol in comparison to a setpoint. To reliably segment both the nucleus and the cytosol, we used our custom interface for the pre-trained SAM75 and implemented a temporal filter to resolve timepoints with low contrast between the intensity of the nucleus and the cytosol (Supplementary Note 2). To illuminate the cells, we chose whole-cell illumination to ensure all optogenetic modules would be maintained in their active state in both the nucleus and cytosol during irradiance.

Model-informed controller design and gain scheduling
First, we measured cell-cell variation in nucleocytoplasmic transport rates to inform our control strategy. In U2OS cells expressing LEXY under a doxycycline-inducible promotor, we observed that the expression level was a major contributor to cell-cell variation, with higher expressing cells having a lower rate of export (Supplementary Fig. 3a). Interestingly, we find that import rates do not correlate with the expression level, as well as the steady-state ratio of intensities between the two compartments (Supplementary Fig. 3b, c). As extended control requires prolonged exposure to low levels of blue light, we also validated that these cells showed only a minimal increase in cell death compared to non-illuminated controls for long-term experiments (Supplementary Fig. 3d–f). After tuning a basic PID controller based on a model of this system (Supplementary Note 3), we were able to control cytosolic intensity to desired setpoints, and the error between measured intensity and setpoint was consistently <10% (Supplementary Fig. 4a–c). However, we found that due to our design constraints aimed at minimising overshoot, these model-derived gains were not universal enough to deal with the non-linear dynamics and cell-cell variation, especially when dynamically changing setpoints across the whole range of cell intensities (the operating range). In these cases, we often observed suboptimal behaviour such as steady-state offsets, depending on the cell (Supplementary Fig. 4d).

Therefore, we improved out controller by including gain scheduling, which adjusts PID gains as a function of a scheduling variable79—in our case, measured intensity—allowing gains to be optimised across the whole operating range. We implemented a gain-scheduled PID controller by first selecting a set of gains for various operating points, using our first-principle model of the system (Supplementary Note 3). We then fine-tuned the controller to find an optimal value-gain matrix. Using this gain-scheduled controller, we were able to control cells for various setpoints within the whole operating range to a high degree of accuracy, overcoming cell-cell variation and stochasticity, and outperforming the standard PID controller across the entire range (Fig. 3c and Supplementary Fig. 4e, f).

Independent control of nuclear and cytosolic protein levels
To demonstrate flexibility in control strategies, and because we envision applications of outcome-driven control of nucleocytoplasmic transport that would benefit from control of nuclear intensity, we also developed a controller to modulate intensity levels in the nucleus (Supplementary Note 3). To compare the two controllers and to show robustness of control, we ran both the cytosol and nucleus controllers in the same cell, repeating three times for each controller (Fig. 3d and Supplementary Movie 6). Both controllers were highly reproducible, and were able to bring intensities to the predefined setpoints with remarkably similar irradiance levels, and with minimal error (Supplementary Fig. 5a–c). Interestingly, we see that the nucleus controller is less noisy than the cytosol controller, likely because the average nucleus intensity is more representative of the real concentration than the average cytosol intensity is for the cytosol, due to the dynamic three-dimensional shape of the cell, the distribution of the marker across the cytosol, and changes of the segmentation through time. We were able to demonstrate long-term repeated control for many setpoints using the nucleus controller, without deviation in error over time (Fig. 3e–g and Supplementary Movie 7). The controller was able to maintain setpoints over extended periods, both when expression was increasing after addition of doxycycline, and after washing out doxycycline, so that expression is more stable (Supplementary Fig. 5d). We were also able to control COS-7 cells with the same implementation, to demonstrate flexibility in experimental design and broader applicability (Supplementary Fig. 5e).

Simultaneous feedback control of protein localisation in multiple cells
Finally, we aimed to simultaneously modulate multiple cells to the same setpoint, showing further controller robustness and overcoming cell-cell variation. To achieve this, we established an alternative spatial light modulator setup that could control multiple cells simultaneously with varying intensities, illuminating individual nuclei with distinct irradiances over time using pulse-width modulation. To avoid unintended activation of cells in close proximity due to light scattering, we chose to illuminate a region in the centre of the cell for these experiments (Supplementary Fig. 6a, b). With this method, we could bring a group of seven cells to the same intensity value (Fig. 4a, b and Supplementary Movie 8). Notably, we could demonstrate that the controller maintains distinct, but constant irradiance levels for each individual cell over time to maintain cells at the desired setpoint. This indicates that future approaches of outcome-driven microscopy could first find these stable values and then require a lower temporal resolution for continued long-term control. To verify that the fluorescence intensity in live-cell imaging is a suitable reporter for the intracellular protein concentration, we controlled multiple cells to different setpoints, fixed the cells on-stage, and visualised the shuttled protein using immunofluorescence (Supplementary Fig. 6c–e). The quantification of immunofluorescence corroborates that the fluorescence intensity measured during live-cell microscopy is a good reporter for protein concentration. Together, these experiments pave the way for the controlled titration of protein levels in nucleus or cytosol, which could for example, be used to explore the dose-response functions of different transcription factors.

Towards outcome-driven microscopy for active control of cell biology
In summary, we have established a proof-of-concept for the robust integration of smart microscopy and optogenetics to overcome cell-cell variation, bring cells to predefined outcomes, and achieve precise, outcome-driven control of microscopy experiments. This approach is also easily applicable to other existing microscope communication platforms that are used for smart microscopy10,80–82, and the modularity allows for a diverse range of experiments, outcomes, and cell types.

While this platform establishes a flexible framework for outcome-driven cell control, practical considerations remain. First, when individually segmenting, analysing and controlling cells, increasing the number of cells controlled in parallel imposes greater computational demands. The computing time required for a full feedback loop scales linearly with the number of cells controlled (Supplementary Note 1). Second, effective outcome-driven control relies on the use of reversible optogenetic modules that must be carefully optimised and characterised for effective controller design. Third, hardware constraints such as the choice of spatial light modulator impacts spatial precision in both 2D and 3D applications, and may limit the ability to modulate light intensity independently and simultaneously across multiple regions and in thicker/heterogeneous samples, crucial for intensity-dependant processes in complex samples.

Advancing the field of smart microscopy by integrating active control of cell biology through optogenetics, we provide a platform for reproducible, cell-cell variation-independent, interrogation of biological processes. This outcome-driven approach can be further expanded into multi-process control and/or 3D environments, in order to provide insight into complex biological interactions and control morphodynamical transitions.

Methods
Cell culture
HT1080 (ATCC, CCL-121), MCF7 (ATCC, HTB-22), U2OS Flp-in T-Rex (a gift from Alessandro Sartori) and COS-7 (ATCC, CRL-1651) cells were cultured in DMEM high glucose medium (Capricorn, DMEM-HPSTA) supplemented with 9% foetal bovine serum (Corning, 35_079_CV) and 1% penicillin/streptomycin (GIBCO, 15140-122) at 37 °C with 5% CO2. All cell lines were routinely screened (every 8–12 weeks) to ensure they were free from mycoplasma contamination.

The HT1080-TIAM cell line was produced for this study using lentiviral transduction. Lentivirus was produced by transfecting 10 cm dishes of HEK293T cells with 15 µg of the transfer plasmid pLenti_TIAM-tagRFP-SSPB-P2A-mVenus-iLID-CAAX (a gift from Mathieu Coppey), together with 10 µg psPAX2 lentivirus packaging plasmid and 5 µg lentivirus envelope plasmid (gifts from Didier Trono, Addgene #12260 and #12259, respectively) using 90 µL 1 mg/mL Polyethylenimine Hydrochloride (Polysciences, 24765). Growth media was harvested 24- and 48-h post-transfection, filtered with a 0.45 µm syringe filter (Corning, 431220) and virus was precipitated in precipitation solution (5× stock solution: 66.6 mM PEG 6000 (Thermo Scientific Chemicals, A17541.0B), 410 mM NaCl (Thermo Scientific Chemicals, 194090010), in ddH2O, pH 7.2). The viral supernatant was then centrifuged for 30 min at 1500×g at 4 °C, and the pellet resuspended in 1× DPBS (Sigma-Aldrich, D8537) for storage at −80 °C. To generate HT1080-TIAM stable line, wild-type cells were seeded to a 24-well plate. Twenty-four hours later, the medium was refreshed with DMEM supplemented with 5 µg/mL polybrene (Sigma-Aldrich, TR-1003), and 5 µL viral suspension was added. The medium was refreshed after 24 h, and cells were first selected in complete medium with 20 µg/mL blasticidin (InvivoGen, 29-11-BL), before sorting for double-positivity with Fluorescence-Activated Cell Sorting (FACS) using a FACSAria™ Fusion Flow Cytometer (BD Biosciences) (Supplementary Fig. 7).

The U2OS-LEXY stable line was generated using flp/frt recombinase-mediated cassette exchange. Maternal U2OS Flp-in T-Rex were transfected with the donor plasmid pCDNA5-NLS-mCherry-LEXY and pOG44 (Invitrogen), carrying Flp recombinase. Isogenic cells stably expressing LEXY from a doxycycline-sensitive promoter were selected by treatment with 200 µg/ml hygromycin B (InvivoGen, ant-hg-5).

For cell migration experiments, HT1080-TIAM cells were seeded to 25 mm coverslips (VWR, 631-0172) coated with 200 µL 25 µg/mL fibronectin (Sigma-Aldrich, F1141) at low density, 6–18 h prior to imaging. 2 mM thymidine (Calbiochem, 6060) was added at the point of seeding to prevent division, and thymidine was kept in the medium for imaging. For nucleocytoplasmic transport experiments, U2OS-LEXY cells were seeded to 25 mm coverslips at medium density, 24 h prior to imaging. 2 µg/mL doxycycline-hyclate (Abcam, ab141091) was added at the point of seeding to induce expression, and doxycycline was kept in the medium for imaging, unless otherwise described. Coverslips were mounted in Attofluor cell chambers (Invitrogen, A7816) for imaging.

For transient transfections when controlling other cell types, cells were seeded to 25 mm coverslips as described above. Twenty-four hours after seeding, MCF7 cells were transfected with 1 µg CMV_TIAM-tagRFP-SSPB-P2A-mVenus-iLID-CAAX (a gift from Mathieu Coppey) and COS-7 cells were transfected with 0.5 µg NLS-mCherry-LEXY (pDN122, a gift from Barbara Di Ventura & Roland Eils, Addgene #72655). FuGENE 6 transfection reagent (Promega, E2691) was used according to manufacturer’s instructions, with 3 µL transfection reagent per 1 µg plasmid DNA.

For phototoxicity experiments, 0.25 µM SYTOX Deep Red Nucleic Acid Stain (Invitrogen, S11380) was added to cells 15 min before imaging. Two stage positions were imaged, one with constant LED illumination and one kept in the dark as a control.

Cloning
The donor vector pCDNA5-NLS-mCherry-LEXY, a mammalian expression vector for LEXY, was derived from NLS-mCherry-LEXY, and pCDNA5/FRT/TO (Invitrogen) by conventional molecular cloning.

Fixation and immunostaining
For quantification of protein localisation after outcome-driven control, cells were fixed on-stage with pre-warmed 4% paraformaldehyde (Thermo Scientific, 28906) for 10 min while maintaining LED illumination. Next, cells were washed with 1× DPBS and permeabilized with 0.2% Triton X-100 (Sigma-Aldrich, X100) for 10 min. Following further washing, blocking was performed with 3% bovine serum albumin (Carl Roth, 8076) for 30 min. Cells were incubated with primary antibody (rabbit anti-RFP; 1:500; Rockland, 600-401-379) for 2 h at room temperature, followed by washing and secondary antibody incubation (goat anti-rabbit Alexa Fluor 488; 1:1000; Thermo Fisher Scientific, A-11034) for 1 h at room temperature. Buffers were prepared in 1× DPBS, while antibodies were diluted in blocking buffer.

Imaging hardware
Imaging was performed using a Nikon Ti dual-turret inverted microscope equipped with a 40× CFI Plan Fluor NA 1.3 oil immersion objective (Nikon, MRH01401), a sample incubator (Tokai-Hit), and a pco.edge cooled sCMOS camera (Excelitas).

For epifluorescence illumination, a pE-4000 LED illumination system (CoolLED) was used as a light source, and ET 514 nm Laser Bandpass (Chroma, 49905) and ET mCherry (Chroma, 49008) filter cubes were placed in the lower filter turret.

For optogenetic illumination, a 470 nm LED (Mightex BLS-LCS-0470-15-22) and a Polygon 400 digital mirror device (Mightex) were equipped for the light source, and a 10/90 beamsplitter (Chroma, 21012) was placed in the upper filter turret. Neutral density filters of optical density 0.5 and 0.9 (ThorLabs, NE205B and NE209B) were also placed into the light path to reduce LED intensity.

For multi-cell control of nucleocytoplasmic transport, a Nikon Ti2-E dual-turret inverted microscope was used, equipped with the above objective, sample incubator, and epifluorescence illumination and filter sets. A Sona-6 Extreme sCMOS camera (Andor, 4BV6U) was equipped. For optogenetic illumination, a pE-800 LED illumination system (CoolLED) was used with a Mosaic 3 digital mirror device (Andor). A neutral density filter of optical density 2.0 (ThorLabs, NE220B) was placed in the light path, and a 10/90 beamsplitter (ThorLabs, BSN10R) was placed in the upper filter turret.

A photodiode power sensor (ThorLabs, S121C sensor with a PM400 console) with built-in irradiance calibration was used to calculate irradiance at the sample plane. For cell migration experiments, images were acquired with a 60 s interval. For nucleocytoplasmic transport experiments, a 15 s interval was used.

Smart microscopy platform for outcome-driven microscopy
Custom Python code was written for control of microscope hardware (Supplementary Note 1) in Python version 3.9.13. We built a modular platform, where an outcome-driven strategy can be defined using separate modules for software-hardware bridging, on-the-fly image analysis, and closed-loop biological control.

In this study, our microscope bridge module uses µ-Manager version 2.083 Nightly Builds 20220930 (Nikon Ti) & 20250310 (Nikon Ti-2E) to control the microscopes and peripheral components, and the Pycro-Manager version 0.19.2 package as a translation layer84. Our on-the-fly analysis module uses Segment Anything75 with custom extensions (Supplementary Note 2), and our closed-loop control module uses either a direct orientation-correction controller (for cell migration), or a PID controller (for nucleocytoplasmic transport) (Supplementary Note 3).

Post-experiment image and data processing
All post-experiment imaging processing was performed using FIJI 1.54f85. Data processing was performed using Python 3.9.13, GraphPad Prism 9.5.0, or Microsoft Excel. GraphPad Prism 9.5.0 and Adobe Illustrator 28.7 were used for data visualisation. MATLAB v.2023b was used for controller design.

Path deviation was determined by calculating the distance between the centroid of the cell and the nearest point on the path (the projected position) at each timepoint. Projected speed was determined by then dividing the distance between projected points by the imaging interval.

Expression levels of cells were calculated as the corrected total cell fluorescence: cell integrated density — (cell area * background mean grey value). Rates of cytosolic intensity change during export or import of LEXY were calculated by measuring cytosolic intensity at 15 s intervals with LED constantly on (10.4 µW/cm2) or off, then fitting to a non-linear one phase decay in GraphPad Prism 9.5.0.

For phototoxicity experiments, SYTOX Deep Red positive cells were counted manually. The total number of cells was estimated by thresholding the image to segment all cells, measuring the total pixel area occupied by cell objects, and dividing this value by the average of 10 cell areas (in pixels).

For comparison between the standard and gain-scheduled PID controllers, mean error at steady-state was calculated by averaging the error from the point where the setpoint was reached. This point is estimated when the derivative of the signal reduced to zero or changed sign. In this way, we clean the error from the transition time.

Nucleus:cytosol intensity ratio, for quantifying of protein localisation after outcome-driven control, was calculated by dividing the (background corrected) mean grey value of the nucleus by that of the cytosol.

Statistics and reproducibility
All information on sample sizes, number of replicates, and statistical analysis is provided in the corresponding figure legends. No statistical method was used to predetermine sample size. No data were excluded from the analyses. The experiments were not randomised, and investigators were not blinded to allocation during experiments and outcome assessment, as this was not necessary for the study.

Peer review rounds
[{"round": 1, "reviewer_comments": {"reviewer_1": "In the paper, the authors describe and implement a closed loop microscopy system to steer biological systems via\noptogenetics in a controlled way depending on a desired spatio temporal pattern. Specifically, the authors describe a\nmodular software framework that is integrated into a microscope setup that allows to detect cells and apply spatially precise\noptogenetic stimulation to induce cell migration in a controlled way. This is achieved by segmenting the cell outline on the fly\nvia a general segmentation model (Segment anything/SAM) and then to selectively illuminate (via a DMD) the cell border\nthat is closest to a predefined setpoint trajectory thus guiding the cell motion along it. The power of this approach is\ndemonstrated on steering multiple (three) cells simultaneously while incorporating live collision avoidance as an additional\ncontrol objective. Finally, the approach is demonstrated on a different task of controlling the relative subcellular expression\nof a protein in the cytosolic and nuclear region of a cell.\nOverall, the paper is well written and the experimental results are quite impressive in that they on original application of the\npowerful idea of closed loop steering of microscopes with on the fly-analysis outside of purely optimizing imaging conditions.\nThe steering of multiple cells in particular is very impressive. As such, I think the paper is a very solid practical\ndemonstration of\na novel application of the concept of smart microscopy that should be of high interest to the community.\nI only have two issues that I think should be addressed in the paper:\n1) The authors write that they provide a \"modular smart microscopy platform\" but I think that the current state of the\nsoftware/repository is more of a demonstration that really a framework that could be easily used by other researchers.\nSpecifically,\n- the modularization of the code happens in a single file (main.py), which makes it hard to see how different parts of the\nsystem relate to each other and could be reused. I would suggest to convert the repository into a more modular library that\ncan be directly installed, rather than a single file\n- some dependencies are missing (eg SAM is imported but not declares as dependency)\n2) The paper could be improved by discussing the limitations of the approach or their experimental demonstration.\nSpecifically,\n- what is the limitation regarding number of cells that can be steered?\n- what is the spatial resolution of the illumination device (eg DMD) required to successfully steer cells?\nminor comments:\n- data figshare link (https://doi.org/10.6084/m9.figshare.26509702) is broken\n- ref SAM [74] arxiv ref should maybe be updated to the published version in ICCV 2023", "reviewer_2": "In this manuscript, the authors present outcome-driven microscopy that combines AI-driven image analysis with optogenetic\ncontrol in a closed-loop configuration. The study demonstrates that cell migration can be continuously guided along a\npredetermined trajectory without manual intervention and that protein expression levels can be modulated with subcellular\nprecision. As stated in the manuscript, closed-loop control combined with optogenetics has been previously explored in both\nin vivo and in vitro by several groups, the contribution of this work lies in its automatic, subcellular control over dynamic\ncellular processes. However, additional experimental evidence is needed to demonstrate precise spatiotemporal control\nover cellular behavior.\nOne of the major challenges in studying cellular dynamics is the inherent stochasticity of protein expression and the\nheterogeneity within cell populations. The implementation of an automatic control system, as described in this study, offers\nthe advantage of capturing a large number of events, which in turn enables a more robust statistical analysis of\nsubpopulation behaviors and dynamic responses. The authors have successfully demonstrated multi-cell manipulation. It is\nreasonable to expect that, by scaling up the system, the number of cells that can be simultaneously monitored and controlled\ncould increase by at least an order of magnitude compared to conventional techniques. Furthermore, by expanding the field\nof view, the system could potentially accommodate even larger cell populations, which would be a significant step forward in\nhigh-throughput dynamic cellular studies.\nOne aspect that requires further clarification is the methodology for optogenetic activation at the subcellular level. The\nmanuscript does not provide detailed information regarding the illumination strategy to target specific cell regions. For\ninstance, it remains unclear whether the area of illumination within the cytosol and nucleus is confined to a specific\nsubregion or if it covers the entire compartment. Given that cellular kinetics may vary depending on the spatial distribution of\nthe light stimulus and considering that cells are inherently three-dimensional structures where a two-dimensional\nillumination pattern might simultaneously affect both cytosolic and nuclear compartments—this ambiguity could have\nsignificant implications for data interpretation.\nAnother point of concern is the reliance on measured intensity values to regulate the system under the assumption that the\nexpression level of the optogenetic construct remains constant. In long-term experiments, phototoxicity is a potential issue\nthat could diminish cellular responsiveness, thereby introducing errors in dynamic measurements. The current\ndemonstration involves only two cycles of intensity control. To better assess the robustness and reliability of the system,\nit would be advantageous to conduct extended, long-term control experiments that evaluate how phototoxic effects and\npotential changes in construct expression influence the system’s performance over time.\nThe performance of the AI model used for real-time analysis is heavily dependent on the training dataset, and in this study,\nthe authors have only validated their approach using two different cell types. This raises the question of whether the\napproach can be applied to other cellular systems. Expanding the experimental validation to include additional cell types\nwould be important for establishing the broader applicability of the system.\nWhile the authors demonstrate controlled titration of protein levels in the nucleus and cytosol based solely on intensity\nmeasurements, it would strengthen their conclusions if these measurements could be corroborated by alternative validation\ntechniques. Furthermore, data presented in Figures 1h and 1i suggest that neither the speed of cell migration nor the\nchanges in speed appear to be significantly dependent on the irradiance. A more detailed discussion is warranted to explain\nthese observations, clarifying whether there are underlying mechanisms or compensatory processes at play that might\naccount for the apparent independence from irradiance."}, "author_response": "Reviewer #1 (Remarks to the Author):\nIn the paper, the authors describe and implement a closed loop microscopy system to steer biological\nsystems via optogenetics in a controlled way depending on a desired spatio temporal pattern.\nSpeci(cid:980)ically, the authors describe a modular software framework that is integrated into a microscope\nsetup that allows to detect cells and apply spatially precise optogenetic stimulation to induce cell\nmigration in a controlled way. This is achieved by segmenting the cell outline on the (cid:980)ly via a general\nsegmentation model (Segment anything/SAM) and then to selectively illuminate (via a DMD) the\ncell border that is closest to a prede(cid:980)ined setpoint trajectory thus guiding the cell motion along it.\nThe power of this approach is demonstrated on steering multiple (three) cells simultaneously while\nincorporating live collision avoidance as an additional control objective. Finally, the approach is\ndemonstrated on a different task of controlling the relative subcellular expression of a protein in the\ncytosolic and nuclear region of a cell.\nOverall, the paper is well written and the experimental results are quite impressive in that they on\noriginal application of the powerful idea of closed loop steering of microscopes with on the (cid:980)ly-\nanalysis outside of purely optimizing imaging conditions. The steering of multiple cells in particular\nis very impressive. As such, I think the paper is a very solid practical demonstration of\na novel application of the concept of smart microscopy that should be of high interest to the\ncommunity. I only have two issues that I think should be addressed in the paper:\n1. The authors write that they provide a \"modular smart microscopy platform\" but I think that the\ncurrent state of the software/repository is more of a demonstration that really a framework that\ncould be easily used by other researchers. Speci(cid:980)ically,\n- the modularization of the code happens in a single (cid:980)ile (main.py), which makes it hard to see\nhow different parts of the system relate to each other and could be reused. I would suggest to\nconvert the repository into a more modular library that can be directly installed, rather than a\nsingle (cid:980)ile\n- some dependencies are missing (eg SAM is imported but not declares as dependency)\n We thank the reviewer for their positive assessment and constructive comments.\nFollowing the reviewer’s suggestion, we have changed the structure of our software\nimplementation to follow a package structure, with a setup.py installation (cid:976)ile that\nincludes all the required libraries. We also renamed the previous main.py as an use-case.\nIt is important to note that the main (cid:976)ile just coordinates the modules, while each module\ncan still be imported and used separately. To further demonstrate this, we have added an\nexample simpli(cid:976)ied implementation to the github repository.\n In addition, we have improved our explanations of dependencies and package structure\nin the readme. Some packages such as SAM, although contained now in the setup.py (cid:976)ile,\nare only a dependency for speci(cid:976)ic modules depending on the implementation.\n2. The paper could be improved by discussing the limitations of the approach or their experimental\ndemonstration. Speci(cid:980)ically,\n- what is the limitation regarding number of cells that can be steered?\n- what is the spatial resolution of the illumination device (eg DMD) required to successfully steer\ncells?\n As recommended by the reviewer, we have added a section to the text to discuss the\nlimitations and practical considerations of our approach (See lines 263-271).\n To address the question about the number of cells that can be controlled, we have\nperformed additional experiments aimed at controlling more cells and exploring\nlimitations of control. This was possible by implementing a new microscope system\nequipped with a DMD module that could cover a larger (cid:976)ield of view, but also\nindependently modulate light intensity for individual cells using pulse-width modulation.\nUsing this implementation, we have now also developed multicellular control for the\nexperiments to control nuclear/cytoplasmic intensities, and demonstrate independent\nintensity-driven control of seven cells (See Figure 4 and lines 236-247).\n With respect to the spatial resolution required to successfully the steer cells, earlier work\nhas already shown that cell guidance can also be achieved with a single spot of focused\nlight (e.g. 10.1038/nature08241 , 10.1371/journal.pbio.3002307). This indicates that the\nspatial resolution of the illumination device is not a factor for these experiments, but we\ndo envisage experiments where this could be important (e.g. precise control of individual\ncells in crowded environments or in the third dimension) as such, we have included this\nin our discussion of practical considerations.\nminor comments:\n- data (cid:976)igshare link (https://doi.org/10.6084/m9.(cid:976)igshare.26509702) is broken\n We thank the reviewer for bringing this to our attention. We have since (cid:976)ixed the above\nlink (for our preprinted version), but have replaced the link with another for the full\ndataset including experiments for this revision (https://doi.org/10.24416/UU01-\nM0U1OT).\n- ref SAM [74] arxiv ref should maybe be updated to the published version in ICCV 2023\n We have updated the reference to SAM as suggested.\nReviewer #2 (Remarks to the Author):\nIn this manuscript, the authors present outcome-driven microscopy that combines AI-driven image\nanalysis with optogenetic control in a closed-loop con(cid:980)iguration. The study demonstrates that cell\nmigration can be continuously guided along a predetermined trajectory without manual\nintervention and that protein expression levels can be modulated with subcellular precision. As\nstated in the manuscript, closed-loop control combined with optogenetics has been previously\nexplored in both in vivo and in vitro by several groups, the contribution of this work lies in its\nautomatic, subcellular control over dynamic cellular processes. However, additional experimental\nevidence is needed to demonstrate precise spatiotemporal control over cellular behavior.\n1. One of the major challenges in studying cellular dynamics is the inherent stochasticity of protein\nexpression and the heterogeneity within cell populations. The implementation of an automatic\ncontrol system, as described in this study, offers the advantage of capturing a large number of\nevents, which in turn enables a more robust statistical analysis of subpopulation behaviors and\ndynamic responses. The authors have successfully demonstrated multi-cell manipulation. It is\nreasonable to expect that, by scaling up the system, the number of cells that can be\nsimultaneously monitored and controlled could increase by at least an order of magnitude\ncompared to conventional techniques. Furthermore, by expanding the (cid:980)ield of view, the system\ncould potentially accommodate even larger cell populations, which would be a signi(cid:980)icant step\nforward in high-throughput dynamic cellular studies.\n We thank the reviewer for the enthusiastic and constructive feedback on our manuscript.\nFor our revised manuscript, we implemented an additional microscope system that\nenabled us to control larger number of cells simultaneously with constant monitoring and\nanalysis (See Figure 4 and lines 236-247).\n2. One aspect that requires further clari(cid:980)ication is the methodology for optogenetic activation at\nthe subcellular level. The manuscript does not provide detailed information regarding the\nillumination strategy to target speci(cid:980)ic cell regions. For instance, it remains unclear whether the\narea of illumination within the cytosol and nucleus is con(cid:980)ined to a speci(cid:980)ic subregion or if it\ncovers the entire compartment. Given that cellular kinetics may vary depending on the spatial\ndistribution of the light stimulus and considering that cells are inherently three-dimensional\nstructures where a two-dimensional illumination pattern might simultaneously affect both\ncytosolic and nuclear compartments—this ambiguity could have signi(cid:980)icant implications for\ndata interpretation.\n We thank the reviewer for raising this point. It is correct to assume that the area of\nillumination has an effect on the kinetics of response to light intensity. Illumination of the\nwhole cell vs nucleus ensures that optogenetic modules are held in the activated state\nthroughout the cell during control. As such, during the development of our method, we\nused global illumination to control protein levels, and have added an explanation of this\nto the main text (See lines 175-177).\n However, for multi-cell control of nuclear intensity, we chose to illuminate the nuclei only,\nin order to prevent unintended illumination due to light scattering when controlling cells\nin close proximity. We have added an explanation of this to the text (See lines 240-242).\nIn both cases, global illumination and nucleus illumination, the control strategy drives the\nconcentration to the desired setpoint without signi(cid:976)icant differences.\n Indeed, a two-dimensional light pattern illuminating just the nucleus is expected to also\naffect some of the cytosolic compartment. However, as we do not derive intrinsic\nbiological parameters from this control, we do not consider this to signi(cid:976)icantly affect our\nconclusions.\n3. Another point of concern is the reliance on measured intensity values to regulate the system\nunder the assumption that the expression level of the optogenetic construct remains constant. In\nlong-term experiments, phototoxicity is a potential issue that could diminish cellular\nresponsiveness, thereby introducing errors in dynamic measurements. The current\ndemonstration involves only two cycles of intensity control. To better assess the robustness and\nreliability of the system, it would be advantageous to conduct extended, long-term control\nexperiments that evaluate how phototoxic effects and potential changes in construct expression\nin(cid:980)luence the system’s performance over time.\n As we normalize the measured intensity values with the dynamic range of the cell, we do\nnot expect changes in (cid:976)luorescence, either due to expression or photobleaching, to\nsigni(cid:976)icantly affect our control. To address this point, we now included extended, long-\nterm control experiments, both with an increased number of setpoints, and maintenance\nof a single setpoint long-term during expected expression increase and stability (See Fig.\n3e-g, Extended data Fig. 5d-e, and lines 228-235).\n We have furthermore examined phototoxicity due to blue-light exposure and found no\nphototoxic effects within the timescales and irradiance levels used in this study (See\nExtended Data Fig. 3d and lines 197-200)\n4. The performance of the AI model used for real-time analysis is heavily dependent on the training\ndataset, and in this study, the authors have only validated their approach using two different cell\ntypes. This raises the question of whether the approach can be applied to other cellular systems.\nExpanding the experimental validation to include additional cell types would be important for\nestablishing the broader applicability of the system.\n The AI model that we used for cell segmentation was a generic pretrained model and is\nnot expected to differ by cell type. However, to validate our segmentation and control\nstrategy on other cell types, we have included demonstration of control of an additional\ncell type for cell migration (See Supplementary Video 5 and lines 146-150), and an\nadditional cell type for nucleocytoplasmic transport (See Extended Data Fig. 5e and lines\n233-235).\n5. While the authors demonstrate controlled titration of protein levels in the nucleus and cytosol\nbased solely on intensity measurements, it would strengthen their conclusions if these\nmeasurements could be corroborated by alternative validation techniques.\n We were unsure how to experimentally address this comment. In our work, we\ndemonstrate that we can dynamically control protein levels in speci(cid:976)ic cellular\ncompartments by controlled exposure to blue light using an approach that is tailored to\nindividual cells to account for variations in overall expression of the LEXY construct. It is\nnot possible to directly measure protein concentration in these assays, as we control live\nsingle cells on the (cid:976)ly in a non-invasive way. Because these processes are controlled\ndynamically using a light microscope, intensity measurements are the only way to assess\ncontroller success.\n Any alternative validation technique that we could conceive to validate protein levels\nwithout intensity-based measurements would involve biochemical assays performed\nafter the experiment, but this poses some problems. First of all, when blue-light exposure\nis aborted the LEXY construct will quickly return to the nucleus, so the desired level is\nonly maintained during the actual microscopy experiment. Second, even if we could\nrapidly (cid:976)ix cells during the experiment and perform subsequent nuclear puri(cid:976)ication and\nbiochemical analysis, this would require bulk experiments controlling all cells in the\nsample, while we can only control a relatively small number of cells compared to the entire\npopulation.\n6. Furthermore, data presented in Figures 1h and 1i suggest that neither the speed of cell\nmigration nor the changes in speed appear to be signi(cid:976)icantly dependent on the irradiance. A\nmore detailed discussion is warranted to explain these observations, clarifying whether there\nare underlying mechanisms or compensatory processes at play that might account for the\napparent independence from irradiance.\n We (cid:976)ind the relationship between irradiance and cell speed very intriguing, and indeed\nexpect additional underlying mechanisms to be in play. As suggested, we have added an\nadditional discussion about this to the main text (See lines 126-132)."}, {"round": 2, "reviewer_comments": {"reviewer_1": "In the revised version the authors have addressed all my original issues and I have no further concerns.", "reviewer_2": "In the revised manuscript, the authors demonstrate simultaneous closed-loop control of seven U2OS-LEXY nuclei, where\nlive cells are guided toward user-defined states in real-time. Such a system may have important applications in the study of\ncell biology. Nevertheless, we still have several concerns about this manuscript.\n1. The authors have employed a nucleus-only illumination strategy to avoid cytosolic activation, but they do not quantify\naxial confinement. A confocal z-stack of a fluorescent calibration slide illuminated through the same optical path—or an\noptical simulation—would reveal the spread of light above and below the focal plane. Such data are particularly important if\nthe technique is to be extended to thicker or more heterogeneous specimens.\n2. Phototoxicity experiments are not so convincing. U2OS-LEXY cells were monitored for morphological changes during six\nhours of repeated activation, but viability was not assessed beyond cell shape, and no assay was performed on the HT1080\ncells used in migration studies. Different cell types can show different sensitivity to blue light. We encourage the authors to\ninclude a standard ROS or live/dead stain for HT1080s and to extend the U2OS assay to twenty-four hours to mimic multi-\nday experiments.\n3. Scalability is another concern in the revised manuscript. Showing control of seven nuclei is a useful proof of concept;\nhowever, readers still have no sense of the practical ceiling. They should provide some key parameters, such as frame-to-\nactuation latency and maximum sustainable frame rate as a function of cell count.\n4. Another concern is the absence of an alternative validation that mCherry fluorescence intensity truly reports intracellular\nprotein concentration over the time scales explored. All quantitative claims rely on this linear mapping. The authors argue\nthat photobleaching correction is enough, yet they provide no biochemical evidence. We suggest that the authors fix a\nsubset of cells at the end of a closed-loop experiment and assay protein levels by Western blot or quantitative\nimmunofluorescence."}, "author_response": "Point-by-point response to the reviewer comments (second round of revision)\nReviewer #1 (Remarks to the Author):\nIn the revised version the authors have addressed all my original issues and I have no further\nconcerns.\n\nRevWieew tehra #n2k t(hRee rmevairekwse tro f othr eth Aeuirt hpoors)it:i ve assessment and constructive comments.\nIn the revised manuscript, the authors demonstrate simultaneous closed-loop control of seven U2OS-\nLEXY nuclei, where live cells are guided toward user-deined states in real-time. Such a system may\nhave important applications in the study of cell biology. Nevertheless, we still have several concerns\nabout this manuscript.\n1. The authors have employed a nucleus-only illumination strategy to avoid cytosolic activation, but\nthey do not quantify axial coninement. A confocal z-stack of a luorescent calibration slide\nilluminated through the same optical path—or an optical simulation—would reveal the spread of\nlight above and below the focal plane. Such data are particularly important if the technique is to be\nextended to thicker or more heterogeneous specimens.\n\nWe would like to clarify that the goal of nucleus-only illumination was not to avoid cytosolic\nactivation, but to limit accidental activation of neighboring cells. Axial coninement could\nindeed be important for more complex 3D samples, but is not very critical in our 2D\nexperiments. Furthermore, all cells are illuminated in the same way, so the controller already\ntakes into account effects like incomplete nucleus illumination or accidental cytosol\nillumination above/below the nucleus. We now include a confocal z-stack of the activation\nvolume in Extended Data Figure 6A and have adjusted our wording in the main text for clarity\n2. P(hlionteost o2x4i9ci-t2y5 0e)x.p eriments are not so convincing. U2OS-LEXY cells were monitored for\nmor phological changes during six hours of repeated activation, but viability was not assessed\nbeyond cell shape, and no assay was performed on the HT1080 cells used in migration studies.\nDifferent cell types can show different sensitivity to blue light. We encourage the authors to include\na standard ROS or live/dead stain for HT1080s and to extend the U2OS assay to twenty-four hours\nto mimic multi-day experiments.\n\nWe have replaced the 6-hour morphological analysis with more robust 24-hour viability\nassessments using live/dead staining. This was performed for both U2OS (Extended Data\nFigure 3d-f, lines 2 06-207) and HT1080 cells (Extended Data Figure 2h–j, lines 255-260). Our\nresults show that optogenetic activation does not strongly affect the live/dead cell ratio over\n3. Scaa l2a4b-ihliotyu ris p aenroiothde.r concern in the revised manuscript. Showing control of seven nuclei is a useful\nproof of concept; however, readers still have no sense of the practical ceiling. They should provide\nsome key parameters, such as frame-to-actuation latency and maximum sustainable frame rate as\na function of cell count.\n\nIn the Supplementary Text (section 1.9), we now provide a discussion on how key\nexperimental parameters scale with the number of cells to be controlled. We also clarify in the\nmain text (lines 279-280) that the computing time scales linearly with the number of cells\n4. Ancoonthtreor lcleodn.c ern is the absence of an alternative validation that mCherry luorescence intensity\ntruly reports intracellular protein concentration over the time scales explored. All quantitative\nclaims rely on this linear mapping. The authors argue that photobleaching correction is enough, yet\nthey provide no biochemical evidence. We suggest that the authors ix a subset of cells at the end of\na closed-loop experiment and assay protein levels by Western blot or quantitative\nimmunoluorescence.\n\nBecause we only control a small subset of all cells presents on the coverslip, Western blotting\nof the lysates from all cells of the coverslip will not be able to assess protein level changes in\nthe few cells that were controlled. To address the reviewer’s concern, we have now performed\nquantitative immunoluorescence analysis on cells that were controlled to speciic levels\nusing optogenetics, as well as neighboring cells that were not controlled (Extended Data\nFigure 6b-d, lines 255-260). This revealed a strong similarity between the nuclear/cytosolic\nratios obtained during live-cell imaging and those obtained from quantitative\nimmunoluorescence."}, {"round": 3, "reviewer_comments": {"reviewer_2": "The authors answered most of our questions. We only have one minor question. The revised manuscript includes a z-stack of the activation volume (Extended Data Fig. 6a), which is presented qualitatively. No quantitative axial spread metric, such as FWHM of the activation field, is reported. A brief quantification would strengthen claims about extensibility to thicker/heterogeneous samples."}, "author_response": "Point-by-point response to the reviewer comments (third round of revision)\nReviewer #2 (Remarks to the Author):\nThe authors answered most of our questions. We have no more questions.\n As requested by the reviewer, we have added a quantification (line profile) of the activation\nfield in Extended Data Figure 6b."}]

Decision letter