Q-learning for real time control of heterogeneous microagent collectives
Authors: Ana Rubio Denniss
Laia Freixas Mateu
Thomas Gorochowski
Sabine Hauert
Venue: ICLR 2022 Submitted (2022)
Keywords: q-learning
reinforcement learning
microsystems
closed-loop control
optical control

Abstract: The effective control of microscopic collectives has many promising applications, from environmental remediation to targeted drug delivery. A key challenge is understanding how to control these agents given their limited programmability, and in many cases heterogeneous dynamics.  The ability to learn control strategies in  real time could allow for the application of robotics solutions to drive collective behaviours towards desired outcomes. Here, we demonstrate Q-learning on the closed-loop Dynamic Optical Micro-Environment (DOME) platform to control the motion of light-responsive Volvox agents. The results show that Q-learning is efficient in autonomously learning how to reduce the speed of agents on an individual basis.

Under review as a conference paper at ICLR 2022
Q- --EAARNING FOR REAL TIME CONTROL OF HETERO- GENEOUS MICROAGENT COLLECTIVES
Anonymous authors Paper under double-blind review
ABSTRACT
The effective control of microscopic collectives has many promising applications, from environmental remediation to targeted drug delivery. A key challenge is understanding how to control these agents given their limited programmability. and in many cases heterogeneous dynamics. The ability to learn control strate- gies in real time could allow for the application of robotics solutions to drive the behaviour of microscopic collectives towards desired outcomes. Here, we demonstrate Q-learning on the closed-loop Dynamic Optical Micro-Environment (DOME) platform to control the motion of light-responsive Volvor agents. The results show that Q-learning is efficient in autonomously learning how to reduce the speed of agents on an individual basis.
INTRODUCTION
The ability to control the behaviour of agents at the microscale or smaller such has implications across fields such as nanomedicine (Hauert & Shatti](20144) and environmental remediation (Wang al. (2019). with possible agent types including micromotors, nanoparticles and bacterial cells. Exerting control at this scale remains challenging however, due in large part to the simplicity and limited programmability of typical microagents. is used control the microagents, here Volvox algae. Machine learning allows for the fine-tuning of the control to each individual Volvax in real-time. Through this, individual models can be learnt that enable optimal motion control, in this case learning how to alternate illumination and relaxation periods to stop the motion of individual Volvox.
this work, an external optical control scheme
Light is a powerful tool at the microscale, capable of forming and breaking bonds [Chen et al. [0188]. powering micromotors (Palagi et al. (2019)), ) and interacting with light sensitive organisms Jekely al. (2008)). Furthermore, the use of spatially structured light offers interaction with agents independently and in parallel (Palagi et al. (2019). making it particularly well suited to the control of collective systems (Mukherjee al. (201 8); Lzquierdo et al. (2018); ; (Schmidt et al. (2019): Deng et al. (2018)). The dynamic nature of light also means that it can be combined with Q-learning to produce rapid and effective and closed-loop control outcomes. This was demonstrated by Muiños- Landin et al., with the use of tabular Q-learming on self- theeooorroopoetticcc microswimmers to achieve navigation in noisy, giid-like environment (Muiños- similarly uses tabular Q-lcarning to influence the dynamics of motile microscale agents using optical interactions, however in this case, each agent performs the learning independently, with significant heterogeneity present among the collective of agents owing to their biological nature. Furthermore, the learning and closed-loop optical control were here implemented on low-cost, open source plat- form, demonstrating the power of this learning process even in instances with limited computational resources.
al. (02T))) The work presented here
Optical control is enacted using the open source DOME platform, a light-weigh device which com- bines digital light projection with microscopy to image a microsystem in real time and provide closed-loop localised light patterning. Given the limited computing power of the DOME, which operates on a Raspberry Pi computer, this work provides an exploration of the potential for the application of Q-learning algorithms in low computational resource environments.
Results show that tabular Q-learning allows us to learn how light may be projected onto Volvax algae in order to maximally reduce their velocity. The state and action space of complex biological

---

Under review as a conference paper at ICLR 2022
system is simplified as run tabular Q-learning experiment, and the learnt values for individual agents used to achieve herding behaviour in living algae.
2
METHODOLOGY
This section introduces the experimental setup for light-based control of Volvox, the simulation environment, and Q-learning methodology applied both in simulation and reality.
2.1 OPTICALLY CONTROLLING Volvex
Volvor are a type of green microscopic algae that exhibit phototactic behaviour. They multi- cellular organisms, with somatic cells that have flagella for locomotion and an eyespot for light perception. These cells allow the Volvor move towards light source (Ueki al. (2010)). This phototactic response is adaptive, meaning that when Volvox comes into contact with light its speed typically reduced for around 2s before adapting to the new light environment and recovering previous velocity (Dreschrr et al. (2010))
In this work, the light response exhibited by Volvor is used a means to regulate the velocity of individual agents by providing spatially localised illumination. To overcome the adaptive nature of the response, illumination must be provided intermittently rather than as a continuous stimuli. Q- learning is therefore applied as a means to determine the optimum cycle length of illumination and relaxation for each agent that results in the largest velocity reduction. The Volvox used here were acquired from Blades Biological UK and are of the species Volvox aureus.
2.2 THE DOME
The experimental part of this work was performed using the DOME (Figure D.. an open source platform for the study and engineering of microagent collectives through spatiotemporal illumina- tion (Denniss et al. (2020). In this device, a closed-loop control scheme is established by linking digital light processing unit to real time imaging and image analysis, enabling the optical micro- environment to be shaped around the evolving system dynamics. The DOME has maximum pro- jection resolution of x0 30 μm, and is thus well suited to illumination of individual Volvox agents, which are around 350-500 um in diameter.
2.3 Q-LEARNING FOR Volvox CONTROL
Due to inherent variability of living algae, in order have adaptable method of control of the Volvox, a reinforcement learning algorithm was required. Because the algae would be controlled using the DOME system, the learning algorithm could not be computationally expensive. Although this could have been circumnavigated by running the algorithm an external computer in commu- nication with the DOME, this work aimed explore the potential for implementing reinforcement learning in limited resource environments. Additionally, maintaining a self-contained computational set up allows for the possibility of operating the system in enclosed conditions, such as within an incubator for live cell study.
For this reason, tabular Q-learning was chosen, instead of more flexible alternatives such as deep Q-learning.
Tabular Q-learning has the restriction of needing discrete action space and state space, but bio- logical systems are inherently continuous. Due to this restriction, the action and state space were defined in a discrete way: The action space consisted on two actions, either illuminate the Volvox, or not. The state space needed represent the amount of light that a Volvox had received.
The Volvox's speed is affected by the amount of light and darkness received. If the state space could be continuous, it would be defined by the amount of time (in milliseconds) that the agent had been illuminated and non-illuminated. Instead of measuring milliseconds, the measurement was discretised using the amount of frames. Since the number of states had be finite, the number of frames of light or darkness could not grow infinitely. However, observation showed that after 10 frames of either illumination or darkness, the agent's behaviour did not change anymore. Because
2

---

Under review as a conference paper at ICLR 2022
Light pattern
Microsystem
Feedback
control
Real-time image analysis
b
Figure I: The DOME platform, shown schematically (a) and pictorially (b). A digital light process- ing projection module is used to controllable pixel grid of by standard light microscopy allow the light patterns be shaped around the evolving dynamics of the microagent system to achieve closed-loop control.
light. Real-time image analysis provided
of this, if an agent hasn't had a change in illumination for over 10 frames, it will be in the same state as if had had the same illumination for 10 frames.
A state was therefore defined by the number of frames for which the agent was subjected to light, the number of frames for which was not subjected to light, and the present light value. The present light value that had light off, then light on. Using this method, the total number states was 242, which was the combination of possible frames (fens between 0 and 10) and frames off (fars. between O and 10) and the light value (I, either ON or OFF). Note that testing all combinations would not be possible in real time, as each trial is real-world experiment involving Volvox reaction. Table [ shows some examples of states with their descriptions. The state index S(fon:of] was calculated as follows
necessary distinguish between state that had light on, then light off, and state
S(fn:foffs fon + 11 soff + 121
Description
f
loff
[
Index
The light was on for 15 frames, then off for 3 frames
10
OFF
43
The light for 7 frames, then off for 4 frames The light was off for frames, then on for 7 frames The light was off for frames, then on for 4 frames
7
4
OFF
51
7
ON
172
4
ON
202
Table 1: Examples of states and their indices. The parameters fou and fors represent the number of frames for which an agent had light on and off respectively, while / describes the present light value given point.
The reward for each state was calculated based the agent's velocity (v) and acceleration (a) at that state. Since the goal was to minimize the magnitude of the velocity, rewards were given for agents with their velocity below threshold, while accelerating agents were penalized. The direction of acceleration and velocity were not considered. Furthermore, because we wanted to minimize the number of light transitions (from to off and vice-versa), states that had more frames and off
3

---

Under review as a conference paper at ICLR 2022
would have higher rewards. The reward function fon: was defined
Son + -5 -1
Sofs
| 0.05 M 2 0.05 and a >
fon-fro
(2)
M 0.05
At each step, the possible actions that could be performed were turn the light either off for each agent, giving an action space size 2. Given this, the Q-table was initialized as an empty matrix with NumStaats 242 and NumActions that action for that state. The Q-table was updated at each step of the learning as the agent explored the environment and different possible states. The Q-learning algorithm stored the previously chosen action (action), and the previous state (s), so as to update the reward at the next iteration.
where each cell encodes the quality of choosing
2.3.1 VOLVOX SIMULATOR
The proposed learning methodology was refined in simulation before use reality. To this end, an agent-based Volvor simulator was built in Python to perform rapid iterations on the control al- gorithms. This simulator replicates way in which Volvox behave in response to light, and was designed such that all code developed was also suitable run on the DOME platform. Volvor agents were modelled based on three assumptions from observation and literature:
Agent velocity is reduced for period of time when coming into contact with light.
After period of time in contact with light, agent velocity recovers.
The duration of the aforementioned two time periods vary from agent to agent.
The simulated agents follow straight line, with probability of them changing direction. This is to replicate the randomness of the Volvox movement. In both the simulator and real world experiments, the passage of time was broken up using the number of elapsed camera frames, allowing continuous measurement to be discretised. Since each Volvox reacts to light in a different way, there exists a pair of values for the number frames with light fuu. and number of frames with light off fort that if repeated continuously, will keep the agent at its minimum speed. To model this light responsive behaviour of the Volvox, a light accumulator model had two local variables, in which the amount of light (az) and darkness (ap) were stored. These variables were bounded between the values of and 20, and increased exponentially with every new frame of light or darkness. The choice of having an exponential increase was to reflect the fact that each frame of light that an agent received, the agent would have more capacity to absorb the light, because it would adapt to its new illumination environment. If any of these variables went above the maximum value, meant that the agent would no longer react to that impulse. In the following equation, indicates the number of consecutive frames of light, and indicates the number of consecutive frames without light.
otherwise
developed. The emulated agent
(L(L)) ch.
ap(tp) = ADHD
The parameters Ap and 2, indicate the rate at which an agent stops reacting to darkness or light, respectively. These values were calculated based on the number of required on and off frames for the agent to stop.
2, = In(20) for
An ln(20) fart
The code used for this simulator is publicly available online bitbuckkt..rg/haaurrlee
4

---

Under review as a conference paper at ICLR 2022
3 RESULTS
3.1
SIMULATION
Initially, the Volvor simulator described in Section 2.3.1 was used to develop and test a learning algorithm for reducing agent velocity.
3.1.1
SINGLE AGENT CONTROL
Qum(Q)
-300
-500
18:03.00 00
18:04:00
18:05.00
18:95.00
18:07.00 .000
18:05.00 00
18:09.00 00
18-10.00
18:1000
18:2:00
18:1000
Time
0.175
0.150
Speed (pxiframe)
0.125
0.100
0.075
0.025
0.000
18:03.00
08004000
18 00.0000
18:06:00
18:07:00
18:08.00 90
18:09.00
18:10.00
18:11:00
16:12:00
18:19:00
18:
Time
Figure 2:
(a) Q-table sum converging after 10 minutes for one simulated Volvox. (b) Speed decreas- ing over time for one simulated Volvox.
The Q-learning algorithm was initially run on the simulator for single agent, attempting reduce agent speed as much as possible for the longest amount of time. The total duration of the experiment was of 10 minutes, representing a typical trial in reality. The simulated agents were programmed to stop when they had received 4 frames of light, and then 3 frames without light. The goal of the Q-learning algorithm was to learn this sequence actions order to stop it. The sum of the Q-table over time (Figure[p) Zpp shows that initially, many of the rewards were negative because the agent was penalized. After minutes however, the table values stagnate as the agent has discovered which states will yield the largest rewards. Accordingly, Figure 2b similarly shows that the agent has variable speed until 5 minutes, at which point the speed stabilizes at low values. The learnt Q-table was analyzed to understand the best actions for the agent, well as the actions that were chosen most frequently in the last 1000 actions.
3.1.2 MULTI-AGENT CONTROL
The same learning process was repeated for system of multiple emulated Volvox agents, replicating tyyiccal experiment. The plot in Figure shows how the speeds of all the detected agents is reduced over time. Consistently across agents, velocity is ultimately reduced after a period variation during which the learning process occurs.
5

---

Under review as a conference paper at ICLR 2022
0.15
Speed
0.10
100000
200000
300000
400000
500000
800000
700000
Episade
positions (pixels)
400
300
200
100
50000
100000
160000
200000
250000
300000
Time (ms)
Figure 3: (a) Speeds of multiple agents over time, as Q-learning attempts to stop them. Speeds reduced over time because the learning algorithm discovers optimal parameters for each agent (b) positions over time of emulated agents during herding, showing that 3 agents stop moving in positions where Y > 328. For both plots, each colour represents different agent. It can be seen that one agent (orange) has not yet learned the illumination pattern required to stop moving.
The velocity control demonstrated above then used explore the possibility of 'herding' the Volvox by attempting to gather agents at the lower part of the screen where 2 328. The agents positioned in this lower half were controlled using the learning algorithm in attempt prevent them moving out of the area, while the other ones were not illuminated, allowing them to move freely. Figure 3 shows that over time, of the agents move to the parts of the screen with high Y coordinate. For the time point shown, the learning algorithm had not yet learnt how to control the agent plotted in orange, hence the position moves up and down the screen continuously while other agents are controlled and kept at the correct position.
3.2 EXPERIMENTAL VALIDATION
The algorithms developed in simulation were then implemented on the DOME for experimental validation with real Volvor agents, with the aim of showing velocity reduction and, if possible, herding. First, three runs with no Q-learning were performed to provide a comparison point. The movement of Volvox was initially observed under no illumination, then under continuous localised illumination. Following this, blinking illumination experiment was run in which light was provided intermittently at for the light without using a complex learning algorithm. For the continuous and blinking illumination, localised light was provided to the Volvox agents positioned in the lower half of the sample, or in terms of image coordinate system, where Y 328. This aimed to recreate the conditions that led to herding outcome in simulation. Following the previous experiments, the Q-learning algorithm was the Volvox, video of which can be found at youtu. Bea.ubbe/UepSSRRGGHM
and fars to reduce the degree to which Volvox were able to adapt to

---

Under review as a conference paper at ICLR 2022
Average speed per Volvox (pixels/ms)
0.05
0.04
0.03
0.02
No illumination
Continuous light
Blinking light
learning
Figure 4: Average speeds in each of the experimental conditions, showing that using Q-learning, there is lower average speed and less variance in the position. For the blinking light condition, an outlier can be seen, represented by circular point.
The speed of the Volvox was compared across the three conditions to understand how effectively each method achieved velocity reduction. The box plot in Figure was created by averaging the speed of each of the detected agents over each of the experiments. Except in the case of no illumination, only Volvox speeds from the illuminated portion of the sample (continuous or blinking) were considered. The plot shows that the Q-learning algorithm maintains the Volvor at lower speed than the other strategies with significantly less variance. Despite blinking light having a low average speed, there an outlier that moves more than 0.05 px/ms, meaning that the algorithm is not good enough to stop all agents. A I-test was performed comparing the Q-learning speed values with each of the other conditions, all three of them showing that the difference was significant (p 0.0003). Similarly, a t-test was done comparing the speeds of the continuous illumination condition with the others, all of which showing that the difference significant (p < 0.03). In all of these conditions, was possible for agents to stop moving, despite not having changes in illumination, due to the randomness of biological systems.
To better understand the qualitative behaviour of learned strategies, two agents that had been tracked over long time period were analysed. For both agents, the Q-tables (Figures 5 and c) converged, with the sum of the table stagnating after some time, and agent speeds were kept under 0.05 px/ms during the whole detection. The first part of the corresponding Q-table for agent A (Figure 5P) shows that the reward is maximum when the light was on for frames, and then off for 4 frames. For agent however, the Q-table outcomes (Figure 5lt) differ from that of agent A, suggesting that each Volvex reacts to light in a different manner.
During the learning phase, the algorithm tried different combinations of actions depending on the state of the Q-table at that moment. In Figure 5p. eight consecutive frames from the camera are shown. In all of them, two agents are detected, but each is illuminated at a different rate to keep the speed as low possible, with continuous analysis of the speed to update the rewards of each state. The agent at the top left starts illuminated (t = Os). then light is turned off for frame (until I == 0.3s), then back on for 2 frames (until = 0.9s), then off for 2 frames (until 1.5s), on for frame, and off again. The agent at the bottom right also starts illuminated (1 = 0s), then light is turned off for frames (until 0.6s), then is turned for three frames (until 1.5s), then back off for two more frames (until / = 2.1s).
4 DISCUSSION
Q-learning is suited in unknown environment, and can be used to understand the best way to control microscopic agents in a way that is independent from the agent type and its characteristics. This means that, although Volox algae employed here model microagent, the tools developed could be adapted to suit other stimuli responsive agents. As with most organic systems, and many inorganic, large degree of heterogeneity exists in the stimuli-responses of Volox agents even within

---

Under review as a conference paper at ICLR 2022
0
400
fremes with light on, then frames with the light off 01--8.5 9.92
1.43
350
3 fremes with light on, then 2 frames with the light off 9.03
1.77
300
light
9.85
0.80
250
3 frames with light on, then S frames with the light off 10.15
1.71
200
3 frames with light on, then 4 frames with the light off 10.54 0.00
150
3 frames light off, then frames the light 1.35
100
light
light
3 frames with light off, then 4 frames with the light 1.07
10.64
05:46:00
06:47:0 00
06:48 so
05:49:00
05:50 oo
05:51. 0:000
05:52:00
05:53:00 00
3 frames with light off, then 5 frames with the light on 8.94
0.00
Time
fo-fremes with light off, then frames with the light on 0.00
a
b
0
light on,
the light 8.85
0.68
the light
3 frames with light off, then 2 frames with the light
sum(Q)
200
3 frames with light off, then 3 frames with the light on 0.00
8:27
frames with light off, then frames with the light on 0.48
9.12
light off,
light 0.81
10.40
frames with light off, then S frames with the light
frames with light off, then 4 frames with the light on 2.25 11.18
05: 05:20:00
05:28:00
O*frames with light off, then frames with the light on 0.00 46
Time
0*frames light off,
light 0.00
d
(i) t=0s
(ii) t=
(iii) t=0.6s
(iv) t=0.9s
(v) |=1.2s
(vi) 1=1.5s
(Vii) [=1.8s
(viii) 1=2.1s
Figure 5: (a) Evolution of the sum of the Q-table for agent A, showing stagnation. (b) Partial Q-table values for agent showing that after 3 frames off, then on, the best action is to turn the light off again. (C) Evolution of the sum of the Q-table for agent B, showing stagnation. (d) Partial Q-table values for agent B, showing that after frames with light off, then frameswith light on, the best action is to keep the light on. (e) Two Volvor illuminated different rates using Q-learning.
8

---

Under review as a conference paper at ICLR 2022
475
Average Y position (pixels)
450
400
375
350
300
275
500000
1000000 Time (ms)
1500000
2000000
Figure 6: The average Y position of all Volver agents over time the Q-learning algorithm alternates between illuminating the top and bottom the screen. Red dashed lines indicate the time point at which the illumination half was switched.
the same population. For this reason, learning individually tuned parameters is powerful in achieving precise control.
In both simulation and experiment, Q-lcarning proved successful strategy for reducing the velocity of emulated and real Volvox agents respectively. The results of the Q-tables differ between simula- tion (Figure[2h) and experiment (Figures and d) in that values are larger in real-world experiments than in simulation. This is possibly because the simulator assumed that stopping the Volvor would be more difficult than it turned out be, meaning that agents were less likely be rewarded. De- spite this, both simulated and real results found that agents do not only have one combination of light on and off that is valid. Instead, there are many combinations that allow to keep a low speed, and Q-learning successfully learns these. In the experimental section, it was found that Q-learning provided the most efficient strategy for slowing the Volvox when compared standard continuous or intermittent illumination patterns. This was demonstrated by the lower average speed, and smaller variance in speeds seen in Figure
In addition to the ability to regulate velocity, herding of agents into particular area was also plored. In simulation, this was found to work well, with positions all but one agent being inside the chosen half by the end of the control period (Figure 3p)). In experiments, preliminary results suggest that the same may be possible. Figure 6 shows the average position for a collection of Volvor agents during the experiment. illuminated region. This experiment suggests that using Q-learning, light could be used to gather agents in an area of the sample despite not being able to directly control their direction. However, due the small agent number (4-5) and the large variance in the natural movement of Volvox, even in the absence of light, further experiments are required to verify this outcome.
time, where the illuminated section of the space was switched three times all cases, the switch occurred when most or all agents had moved to the
The potential control entire microagent collective in parallel could also allow for exploration of swarm behaviours and control strategies at the microscale. Broadly, swarm system is one in which agents are able to collectively perform actions that are beyond the capabilities of an individual, typically facilitated through local interactions (Brambilla et (013)). Unlike in macroscale swarm engineering, microagents cannot be straightforwardly programmed with interaction rules, rather agents must typically interact through physical means such as chemical signalling. This requires the design and fabrication of highly complex agents, something that can costly and consumin... Given this, the development of generic microswarm control strategies could be crucial in informing the design of these intelligent micro/nanoagents, efficiently directing the production process to best suit given application.

---

Under review as a conference paper at ICLR 2022
Overall, this work suggests that tabular Q-learning is useful tool to efficiently learn microagent control strategies in real-time on platforms with limited computational resources, as it is not as computationally expensive as other more complex learning algorithms, and is suited to an unknown environment. The use here of the DOME as low-cost, open source platform also significant in widening accessibility to similar control techniques.
FUTURE WORK
The primary goal of this project was to demonstrate the potential of using Q-learning to control individual agents within complex, living biological system in real time. In line with this, a con- trol strategy developed using tabular Q-learning was found to be effective compared to non-learning baselines in regulating the motion of these biological agents. In future work, comparison with alternative learning- based algorithms would be informative in optimising the control process. Fur- thermore, due to the on-board, lightweight nature of the computational strategy developed here there is the potential to apply similar schemes to live mammalian cell environments through operation in- side an incubator environment, or even as miniaturised wearable medical device. This could allow exploration of learning based control for cell collectives such as tumours or healing wounds.
ACKNOWLEDGMENTS
This work supported by an EPSRC DTP scholarship (A.R.D) and the EPSRC TAS pump prim- ing fund (A.R.D, S.H., T.E.G.).
REFERENCES
Manuele Brambilla, Eliseo Ferrante, Mauro Birattari, and Marco Dorigo. Swarm roboties: review from the swarm engineering perspective. Swarm Intelligence. 7(1):1-41) 2013.
Yihuang Chen, Zewei Wang, Yanjie He, Young Jun Yoon, Jaehan Jung, Guangzhao Zhang, and Zhiqun Lin. Light-enabled reversible self-assembly and tunable optical properties of stable hairy nanoparticles. Proceedings of the National Academy of Sciences, E15(7)E1391-114000 2018.
Zhuoyi Deng, Fangzhi Mou, Shaowen Tang, Leilei Xu, Ming Luo, and Jianguo Guan. Swarming and collective migration of micromotors under near infrared light. Applied Materials Today, 13: 45-53, 2018.
Ana Rubio Denniss, Thomas E. Gorochowski, and Sabine Hauert. An open platform for high- resolution light-based control of mieroscopic collectives. bioRxiv, 2020.
Knut Drescher, Raymond E Goldstein, and Idan Tuval. Fidelity of adaptive phototaxis. Proceedings of the National Academy of Sciences, 107(25):11111-1117 2010.
Sabine Hauert and Sangeeta N Bhatia. Mechanisms of cooperation in cancer nanomedicine: towards systems nanotechnology. Trends in Biotechnology, 32(9):448-455, 2014.
Emiliano Izquierdo, Theresa Quinkler, and Stefano De Renzis. Guided morphogenesis through optogenetic activation of rho signalling during early drosophila embryogenesis. Nature commu- nications, 111: 2018
Gáspár Jékely, Julien Colombelli, Harald Hausen, Keren Guy, Ersst Stelzer, François Nédélec, and Detlev Arendt. Mechanism of phototaxis in marine zooplankton. Nature, 456(7220):395-39,, 2008.
S. Muiños- A. Fischer, V. Holubec, and F. Cichos. Reinforcement learning with artificial microswimmers. Science Robotics, 6(52), 2021.
Manisha Mukherjee, Yidan Hu, Chuan Hao Tan, Scott A Rice, and Bin Cao. Engineering light- responsive, quorum quenching biofilm to mitigate biofouling on water purification membranes. Science advances, 4 4(12)eaau 1459, 2018.
10

---

Under review as a conference paper at ICLR 2022
Stefano Palagi, Dhruv P Singh, and Peer Fischer. Light-controlled micromotors and soft micro- robots. Advanced Optical Materials, 7(16):1900370, 2019.
Falko Schmidt, Benno Liebchen, Hartmut Löwen, and Giovanni Volpe. Light-controlled assembly of active colloidal molecules. The Journal of chemical physics, 150(9):094905, 2019.
Noriko Ueki, Shigeru Matsunaga, Isao Inouye, and Armin Hallmann, How 5000 independent rowers coordinate their strokes in order to row into the sunlight: Phototaxis in the multicellular green alga volvox. BMC biology, 8(1):1-21, 2010.
Linlin Wang, Andrea Kaeppler, Dieter Fischer, and Juliane Simmchen. Photocatalytic tio2 micro- motors for removal of microplastics and suspended matter. ACS applied materials & interfaces, 11(66):32937-32244, 2019.


Reviews:
[{"review_id": "tA2RcLCNGw2", "reviewer": "Reviewer_GPp7", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": " Strength:\n* The paper is  clear to understand on a high level, given the application domain. The authors motivate the use case well, justifying the case for applying Q learning for this task.\n* Its (to the best of the reviewers knowledge)a novel application of Reinforcement Learning\n\nWeakness:\n\nQuality of work\n\n *  This is an area for improvement. The level of contribution could be improved. An example would be adding other baselines apart from Q learning. The choice of baselines, which seem like hardcoded policy(random, blinking etc), might not be a fair comparison to the Q learning agent.\n\nOther comments:\n * \"The Q-learning method requires having a finite number of states and actions, and stores a Q-table\n with as many rows as states, and as many columns as actions \"-> It'll be good to clarify this as tabular Q learning. While this is correct, recent advances typing Deep Learning with Reinforcement Learning have allowed networks to generalize from a large state space which is infeasible to be stored in tabular format. In this context,  an interesting follow up would be to combine the pixel values of the microscopic image with the structured state information.\n\n* While it intuitively makes sense on penalizing velocity, acceleration, perhaps more background can be provided on the motivation behind this. Does the direction of velocity also matter or only the absolute magnitude is penalized?\n\n* How well does the dynamics of the simulated environment map to the real world? The approximation seems reasonable, but it would seem that an agent would be able to learn an optimal policy, ie there is no stochasticity, randomness that would add complexity to the environment. \n\n*  Is it possible for the speed to be 0 with continuous illumination?\n\n* If the agent stays in either \"light\"or \"dark\" state for more than 10 frames at a time, how will that be characterized? Is that a feasible scenario?\n", "summary_of_the_paper": "The given work discusses the use of Q learning to control the motion of a light responsive Volvox agent. They also develop a simulation environment, providing an empirical estimate of  the dynamics of the system. They evaluateon both single and multi-agent control. The proposed tabular Q learning agent was able to achieve superior performance(ie slower speed in this setup) compared to the baselines proposed.", "summary_of_the_review": "The paper provides the application of Tabular Q based learning on a a novel application. The author's also develop a simulation of the environment for to test this policy. The paper is reasonably well written, clearly organized. However, the contributions, significance and empirical evaluation are limited. Tabular Q learning is a well studied concept. Only a single learning algorithm is evaluated, while the choice of baselines could be expanded to other agent types.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rQN3BfpauNL", "reviewer": "Reviewer_BtTc", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The main strengths of this paper:\n\n(1) It applies machine learning to real world problems. Such inter-discipline studies are highly encouraged imho, since they demonstrate the real power of machine learning outside standard benchmarks and theories. \n\n(2) It demonstrates that the learned Q table can regulate the motion of the algae significantly better than human designed baselines. \n\n(3) Experiments are conducted in both  simulated and real world environments.\n\nThe main weakness of this paper:\n\n(1) Overall presentation of this paper can be improved.  For example, the state space of the problem is not clearly described and involves some confusing details. The authors seem to use past light on/off history as the state space. From my understanding it is equivalent to concatenating applied actions (turning on light or off)  as the state space. Is that the case? Also, the authors write \"Additionally, if no change in state was detected after 10 consecutive frames in either light or darkness, the state would no longer change.\" What does this mean? How does it change the state space?\n\n(2) Given how the authors design the state space, I think there is a more continuous way to formulate the state space: the micro-agent's velocity as the state, and on-off of light as the action. In this way we will have an alternative formulation. The authors can still discretize the velocity space if they want to use Q-table, but they can also use DNN to represent the Q network. Has this been tried? \n\n(3) No videos etc to demonstrate the real world experiment. Videos will help the readers, who may not be from this field, to understand better. \n", "summary_of_the_paper": "In this paper, the authors use machine learning to control the velocity/motion of a type of microorganisms, the Volvox algae. The algae's velocity will react to light in a non trivial, adaptive way,  and the authors use Q learning to regulate the algae's velocity. Experiments are conducted  in both simulator and real world. The authors demonstrate that after learning, the Q table method can reduce the motion of this micro-agents to be almost stationary, significantly better than baselines.", "summary_of_the_review": "In summary, the authors apply Q-learning to solve a real world control problem for microorganisms. The approach is inter-disciplinary, though the method  (Q table) is well known. The authors demonstrated compiling results in both simulator and in the real world. ", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "SuLEzor7c6d", "reviewer": "Reviewer_QBsR", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "In the tabular Q-learning instantiation, the state space was defined by the amount of consecutive light that an agent had received measured by the number of frames instead of time. The reward is based on agents’ velocity and acceleration The Q-table was initialized as an empty matrix with 242 states and 2 actions, where each cell encodes the quality of choosing that action for that state. \n\nThe paper presents an application of tabular Q-learning to a new and interesting environment (DOME). As such the main contribution is the formulation of states, actions, and rewards to the well-known Q-learning algorithm. As such I am concerned that the scope of the paper is suitable since the contribution mostly focuses on the environment rather than the learning algorithm itself.\n\nComments that could help improve the paper:\n\n- The paper also presents a simulator. It would be good if it were made available to the public.\nWhile the abstract describes the learning process for a collective, the learning actually works on a per-agent level. It would help if this could be clarified earlier on.\n\n- To strengthen the paper, comparisons to other competitive algorithms and baselines are needed. Currently, no comparison algorithms, no comparable baselines, and no ablations are provided\n\n- At times, the paper reads like a report that is too focused on the engineering details of the DOME environment. I would suggest moving some of these into an appendix and focusing on a few main problems that you are solving differently than the state of the art. Then show evidence that your novel solution is better than the state-of-the-art.\n\n- I recommend clearly stating the contributions of the paper\n\nMinor:\n\n- Figure 2 caption: “Q-table converging after 10 minutes for one simulated Volvox” I am not sure whether the table has converged. Yes, all the agents have stopped after 5min, so the table doesn’t change much anymore. But that also means that the state part of the table containing low light visitation regions doesn’t get visited anymore.\n\n- “The DOME operates on a Raspberry Pi computer, meaning that the reinforcement learning algorithms developed here, specifically Q-learning, can not be computationally expensive.“ Why could you not run the learning algorithm on an external computer that communicates to the Raspberry Pi?\n\n- Consider changing writing from passive to active voice\n", "summary_of_the_paper": "The authors propose learning control strategies in real-time for agent collectives.\nThey demonstrate the result of tabular Q-learning on a closed-loop Dynamic Optical Micro-Environment (DOME) platform to control the motion of light-responsive Volvox agents. Specifically, Q-learning allows learning how light may be projected onto Volvox algae to maximally reduce their velocity. ", "summary_of_the_review": "The paper describes the application of tabular Q-learning to the DOME environment. I am worried that the novelty and significance of the algorithmic contributions in this paper are too focused on the DOME environment to apply to other areas of learning. Competitive baselines, and a comparison to state-of-the-art are lacking as well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper applies and evaluates the use of Q-learning for the control of microscopic collectives of Volvox algae. 

While the application is indeed very cool and potentially impactful, the paper has no theoretical contribution to the field of machine learning as it consists of an empirical evaluation of an existing (and well-established) algorithm.

The reviewers agree on the importance of the application, reported concerns about the current manuscript. In particular:
- Reviewers QBsR and GPp7 suggested including additional comparisons to other learning algorithms
- Reviewers QBsR and BtTc also suggested improving the writing

Overall, I agree with the reviewers that the current manuscript has a lot of potentials, but it could benefit from additional work. 
Please carefully consider and incorporate the feedback received from the reviewers. Personally, I think that presenting a more sharp message and clearer insights would further increase the quality of exposition and help to make a stronger case for why this manuscript is relevant to the larger ML community.
Decision: Reject