Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors propose learning control strategies in real-time for agent collectives.
They demonstrate the result of tabular Q-learning on a closed-loop Dynamic Optical Micro-Environment (DOME) platform to control the motion of light-responsive Volvox agents. Specifically, Q-learning allows learning how light may be projected onto Volvox algae to maximally reduce their velocity. 
main_review: In the tabular Q-learning instantiation, the state space was defined by the amount of consecutive light that an agent had received measured by the number of frames instead of time. The reward is based on agents’ velocity and acceleration The Q-table was initialized as an empty matrix with 242 states and 2 actions, where each cell encodes the quality of choosing that action for that state. 

The paper presents an application of tabular Q-learning to a new and interesting environment (DOME). As such the main contribution is the formulation of states, actions, and rewards to the well-known Q-learning algorithm. As such I am concerned that the scope of the paper is suitable since the contribution mostly focuses on the environment rather than the learning algorithm itself.

Comments that could help improve the paper:

- The paper also presents a simulator. It would be good if it were made available to the public.
While the abstract describes the learning process for a collective, the learning actually works on a per-agent level. It would help if this could be clarified earlier on.

- To strengthen the paper, comparisons to other competitive algorithms and baselines are needed. Currently, no comparison algorithms, no comparable baselines, and no ablations are provided

- At times, the paper reads like a report that is too focused on the engineering details of the DOME environment. I would suggest moving some of these into an appendix and focusing on a few main problems that you are solving differently than the state of the art. Then show evidence that your novel solution is better than the state-of-the-art.

- I recommend clearly stating the contributions of the paper

Minor:

- Figure 2 caption: “Q-table converging after 10 minutes for one simulated Volvox” I am not sure whether the table has converged. Yes, all the agents have stopped after 5min, so the table doesn’t change much anymore. But that also means that the state part of the table containing low light visitation regions doesn’t get visited anymore.

- “The DOME operates on a Raspberry Pi computer, meaning that the reinforcement learning algorithms developed here, specifically Q-learning, can not be computationally expensive.“ Why could you not run the learning algorithm on an external computer that communicates to the Raspberry Pi?

- Consider changing writing from passive to active voice

summary_of_the_review: The paper describes the application of tabular Q-learning to the DOME environment. I am worried that the novelty and significance of the algorithmic contributions in this paper are too focused on the DOME environment to apply to other areas of learning. Competitive baselines, and a comparison to state-of-the-art are lacking as well.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this paper, the authors use machine learning to control the velocity/motion of a type of microorganisms, the Volvox algae. The algae's velocity will react to light in a non trivial, adaptive way,  and the authors use Q learning to regulate the algae's velocity. Experiments are conducted  in both simulator and real world. The authors demonstrate that after learning, the Q table method can reduce the motion of this micro-agents to be almost stationary, significantly better than baselines.
main_review: The main strengths of this paper:

(1) It applies machine learning to real world problems. Such inter-discipline studies are highly encouraged imho, since they demonstrate the real power of machine learning outside standard benchmarks and theories. 

(2) It demonstrates that the learned Q table can regulate the motion of the algae significantly better than human designed baselines. 

(3) Experiments are conducted in both  simulated and real world environments.

The main weakness of this paper:

(1) Overall presentation of this paper can be improved.  For example, the state space of the problem is not clearly described and involves some confusing details. The authors seem to use past light on/off history as the state space. From my understanding it is equivalent to concatenating applied actions (turning on light or off)  as the state space. Is that the case? Also, the authors write "Additionally, if no change in state was detected after 10 consecutive frames in either light or darkness, the state would no longer change." What does this mean? How does it change the state space?

(2) Given how the authors design the state space, I think there is a more continuous way to formulate the state space: the micro-agent's velocity as the state, and on-off of light as the action. In this way we will have an alternative formulation. The authors can still discretize the velocity space if they want to use Q-table, but they can also use DNN to represent the Q network. Has this been tried? 

(3) No videos etc to demonstrate the real world experiment. Videos will help the readers, who may not be from this field, to understand better. 

summary_of_the_review: In summary, the authors apply Q-learning to solve a real world control problem for microorganisms. The approach is inter-disciplinary, though the method  (Q table) is well known. The authors demonstrated compiling results in both simulator and in the real world.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The given work discusses the use of Q learning to control the motion of a light responsive Volvox agent. They also develop a simulation environment, providing an empirical estimate of  the dynamics of the system. They evaluateon both single and multi-agent control. The proposed tabular Q learning agent was able to achieve superior performance(ie slower speed in this setup) compared to the baselines proposed.
main_review:  Strength:
* The paper is  clear to understand on a high level, given the application domain. The authors motivate the use case well, justifying the case for applying Q learning for this task.
* Its (to the best of the reviewers knowledge)a novel application of Reinforcement Learning

Weakness:

Quality of work

 *  This is an area for improvement. The level of contribution could be improved. An example would be adding other baselines apart from Q learning. The choice of baselines, which seem like hardcoded policy(random, blinking etc), might not be a fair comparison to the Q learning agent.

Other comments:
 * "The Q-learning method requires having a finite number of states and actions, and stores a Q-table
 with as many rows as states, and as many columns as actions "-> It'll be good to clarify this as tabular Q learning. While this is correct, recent advances typing Deep Learning with Reinforcement Learning have allowed networks to generalize from a large state space which is infeasible to be stored in tabular format. In this context,  an interesting follow up would be to combine the pixel values of the microscopic image with the structured state information.

* While it intuitively makes sense on penalizing velocity, acceleration, perhaps more background can be provided on the motivation behind this. Does the direction of velocity also matter or only the absolute magnitude is penalized?

* How well does the dynamics of the simulated environment map to the real world? The approximation seems reasonable, but it would seem that an agent would be able to learn an optimal policy, ie there is no stochasticity, randomness that would add complexity to the environment. 

*  Is it possible for the speed to be 0 with continuous illumination?

* If the agent stays in either "light"or "dark" state for more than 10 frames at a time, how will that be characterized? Is that a feasible scenario?

summary_of_the_review: The paper provides the application of Tabular Q based learning on a a novel application. The author's also develop a simulation of the environment for to test this policy. The paper is reasonably well written, clearly organized. However, the contributions, significance and empirical evaluation are limited. Tabular Q learning is a well studied concept. Only a single learning algorithm is evaluated, while the choice of baselines could be expanded to other agent types.
review_title: null

Meta-review: This paper applies and evaluates the use of Q-learning for the control of microscopic collectives of Volvox algae. 

While the application is indeed very cool and potentially impactful, the paper has no theoretical contribution to the field of machine learning as it consists of an empirical evaluation of an existing (and well-established) algorithm.

The reviewers agree on the importance of the application, reported concerns about the current manuscript. In particular:
- Reviewers QBsR and GPp7 suggested including additional comparisons to other learning algorithms
- Reviewers QBsR and BtTc also suggested improving the writing

Overall, I agree with the reviewers that the current manuscript has a lot of potentials, but it could benefit from additional work. 
Please carefully consider and incorporate the feedback received from the reviewers. Personally, I think that presenting a more sharp message and clearer insights would further increase the quality of exposition and help to make a stronger case for why this manuscript is relevant to the larger ML community.
Decision: Reject