Q-learning for real time control of heterogeneous microagent collectives

1 INTRODUCTION . The ability to control the behaviour of agents at the microscale or smaller such has implications across fields such as nanomedicine ( Hauert & Bhatia ( 2014 ) ) and environmental remediation ( Wang et al . ( 2019 ) ) , with possible agent types including micromotors , nanoparticles and bacterial cells . Exerting control at this scale remains challenging however , due in large part to the simplicity and limited programmability of typical microagents . In this work , an external optical control scheme is used control the microagents , here Volvox algae . Machine learning allows for the fine-tuning of the control to each individual Volvox in real-time . Through this , individual models can be learnt that enable optimal motion control , in this case learning how to alternate illumination and relaxation periods to stop the motion of individual Volvox . Light is a powerful tool at the microscale , capable of forming and breaking bonds [ Chen et al . ( 2018 ) ] , powering micromotors ( Palagi et al . ( 2019 ) ) , and interacting with light sensitive organisms ( Jékely et al . ( 2008 ) ) . Furthermore , the use of spatially structured light offers interaction with agents independently and in parallel ( Palagi et al . ( 2019 ) ) , making it particularly well suited to the control of collective systems ( Mukherjee et al . ( 2018 ) ; Izquierdo et al . ( 2018 ) ; Schmidt et al . ( 2019 ) ; Deng et al . ( 2018 ) ) . The dynamic nature of light also means that it can be combined with Q-learning to produce rapid and effective and closed-loop control outcomes . This was demonstrated by MuiñosLandin et al. , with the use of tabular Q-learning on self-thermophoretic microswimmers to achieve navigation in a noisy , grid-like environment ( Muiños-Landin et al . ( 2021 ) ) . The work presented here similarly uses tabular Q-learning to influence the dynamics of motile microscale agents using optical interactions , however in this case , each agent performs the learning independently , with significant heterogeneity present among the collective of agents owing to their biological nature . Furthermore , the learning and closed-loop optical control were here implemented on a low-cost , open source platform , demonstrating the power of this learning process even in instances with limited computational resources . Optical control is enacted using the open source DOME platform , a light-weight device which combines digital light projection with microscopy to image a microsystem in real time and provide closed-loop localised light patterning . Given the limited computing power of the DOME , which operates on a Raspberry Pi computer , this work provides an exploration of the potential for the application of Q-learning algorithms in low computational resource environments . Results show that tabular Q-learning allows us to learn how light may be projected onto Volvox algae in order to maximally reduce their velocity . The state and action space of a complex biological system is simplified so as to run tabular Q-learning experiment , and the learnt values for individual agents used to achieve herding behaviour in living algae . 2 METHODOLOGY . This section introduces the experimental setup for light-based control of Volvox , the simulation environment , and Q-learning methodology applied both in simulation and reality . 2.1 OPTICALLY CONTROLLING Volvox Volvox are a type of green microscopic algae that exhibit phototactic behaviour . They are multicellular organisms , with somatic cells that have flagella for locomotion and an eyespot for light perception . These cells allow the Volvox to move towards a light source ( Ueki et al . ( 2010 ) ) . This phototactic response is adaptive , meaning that when a Volvox comes into contact with light its speed is typically reduced for around 2s before adapting to the new light environment and recovering previous velocity ( Drescher et al . ( 2010 ) ) . In this work , the light response exhibited by Volvox is used as a means to regulate the velocity of individual agents by providing spatially localised illumination . To overcome the adaptive nature of the response , illumination must be provided intermittently rather than as a continuous stimuli . Qlearning is therefore applied as a means to determine the optimum cycle length of illumination and relaxation for each agent that results in the largest velocity reduction . The Volvox used here were acquired from Blades Biological UK and are of the species Volvox aureus . 2.2 THE DOME . The experimental part of this work was performed using the DOME ( Figure 1 ) , an open source platform for the study and engineering of microagent collectives through spatiotemporal illumination ( Denniss et al . ( 2020 ) ) . In this device , a closed-loop control scheme is established by linking a digital light processing unit to real time imaging and image analysis , enabling the optical microenvironment to be shaped around the evolving system dynamics . The DOME has a maximum projection resolution of 30×30 µm , and is thus well suited to illumination of individual Volvox agents , which are around 350–500 µm in diameter . 2.3 Q-LEARNING FOR Volvox CONTROL Due to inherent variability of living algae , in order to have an adaptable method of control of the Volvox , a reinforcement learning algorithm was required . Because the algae would be controlled using the DOME system , the learning algorithm could not be computationally expensive . Although this could have been circumnavigated by running the algorithm an external computer in communication with the DOME , this work aimed to explore the potential for implementing reinforcement learning in limited resource environments . Additionally , maintaining a self-contained computational set up allows for the possibility of operating the system in enclosed conditions , such as within an incubator for live cell study . For this reason , tabular Q-learning was chosen , instead of more flexible alternatives such as deep Q-learning . Tabular Q-learning has the restriction of needing a discrete action space and state space , but biological systems are inherently continuous . Due to this restriction , the action and state space were defined in a discrete way : The action space consisted on two actions , either to illuminate the Volvox , or not . The state space needed to represent the amount of light that a Volvox had received . The Volvox ’ s speed is affected by the amount of light and darkness received . If the state space could be continuous , it would be defined by the amount of time ( in milliseconds ) that the agent had been illuminated and non-illuminated . Instead of measuring milliseconds , the measurement was discretised using the amount of frames . Since the number of states had to be finite , the number of frames of light or darkness could not grow infinitely . However , observation showed that after 10 frames of either illumination or darkness , the agent ’ s behaviour did not change anymore . Because of this , if an agent hasn ’ t had a change in illumination for over 10 frames , it will be in the same state as if it had had the same illumination for 10 frames . A state was therefore defined by the number of frames for which the agent was subjected to light , the number of frames for which it was not subjected to light , and the present light value . The present light value was necessary to distinguish between a state that had light on , then light off , and a state that had light off , then light on . Using this method , the total number of states was 242 , which was the combination of possible frames on ( fon , between 0 and 10 ) and frames off ( fo f f , between 0 and 10 ) and the light value ( l , either ON or OFF ) . Note that testing all combinations would not be possible in real time , as each trial is a real-world experiment involving a Volvox reaction . Table 1 shows some examples of states with their descriptions . The state index S ( fon , o f f , l ) was calculated as follows S ( fon , fo f f , l ) = fon +11∗ fo f f +121∗ l. ( 1 ) The reward for each state was calculated based on the agent ’ s velocity ( v ) and acceleration ( a ) at that state . Since the goal was to minimize the magnitude of the velocity , rewards were given for agents with their velocity below a threshold , while accelerating agents were penalized . The direction of acceleration and velocity were not considered . Furthermore , because we wanted to minimize the number of light transitions ( from on to off and vice-versa ) , states that had more frames on and off would have higher rewards . The reward function R ( v , a , fon , fo f f ) was defined as R ( v , a , fon , fo f f ) = { fon + fo f f |v| < 0.05 −5 |v| ≥ 0.05 and a > 0 −1 |v| ≥ 0.05 } . ( 2 ) At each step , the possible actions that could be performed were to turn the light either on or off for each agent , giving an action space of size 2 . Given this , the Q-table was initialized as an empty matrix with NumStates= 242 and NumActions= 2 , where each cell encodes the quality of choosing that action for that state . The Q-table was updated at each step of the learning as the agent explored the environment and different possible states . The Q-learning algorithm stored the previously chosen action ( action ) , and the previous state ( s ) , so as to update the reward at the next iteration . 2.3.1 VOLVOX SIMULATOR . The proposed learning methodology was refined in simulation before use in reality . To this end , an agent-based Volvox simulator was built in Python to perform rapid iterations on the control algorithms . This simulator replicates the way in which Volvox behave in response to light , and was designed such that all code developed was also suitable to run on the DOME platform . Volvox agents were modelled based on three assumptions from observation and literature : • Agent velocity is reduced for period of time when coming into contact with light . • After a period of time in contact with light , agent velocity recovers . • The duration of the aforementioned two time periods vary from agent to agent . The simulated agents follow a straight line , with a probability of them changing direction . This is to replicate the randomness of the Volvox movement . In both the simulator and real world experiments , the passage of time was broken up using the number of elapsed camera frames , allowing an otherwise continuous measurement to be discretised . Since each Volvox reacts to light in a different way , there exists are a pair of values for the number frames with light on fon , and number of frames with light off fo f f that if repeated continuously , will keep the agent at its minimum speed . To model this light responsive behaviour of the Volvox , a light accumulator model was developed . The emulated agent had two local variables , in which the amount of light ( aL ) and darkness ( aD ) were stored . These variables were bounded between the values of 1 and 20 , and increased exponentially with every new frame of light or darkness . The choice of having an exponential increase was to reflect the fact that at each frame of light that an agent received , the agent would have more capacity to absorb the light , because it would adapt to its new illumination environment . If any of these variables went above the maximum value , it meant that the agent would no longer react to that impulse . In the following equation , tL indicates the number of consecutive frames of light , and tD indicates the number of consecutive frames without light . aL ( tL ) = eλL∗tL aD ( tD ) = eλD∗tD The parameters λD and λL indicate the rate at which an agent stops reacting to darkness or light , respectively . These values were calculated based on the number of required on and off frames for the agent to stop . λL = ln ( 20 ) ∗ 1 fon λD = ln ( 20 ) ∗ 1 fo f f The code used for this simulator is publicly available online at bitbucket.org/hauertlab .

Summary: The given work discusses the use of Q learning to control the motion of a light responsive Volvox agent. They also develop a simulation environment, providing an empirical estimate of  the dynamics of the system. They evaluateon both single and multi-agent control. The proposed tabular Q learning agent was able to achieve superior performance(ie slower speed in this setup) compared to the baselines proposed.