Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: The given work develops a fish simulation motivated by solving real world problems in the fish farming industry such as over and underfeeding. Through this work, they wish to replicate fish schooling behaviour with higher fidelity, by leveraging Deep Reinforcement Learning to model this. The authors argue that this resulting setup could then be used for training computer vision models to accurately count varied fish species in the cages.
strengths_and_weaknesses: * Overall, the work is well written, well organized and clear to follow. It reasonably describes the past work in the field, stating how the contributions are distinct

* The authors motivate the problem statement well, on how this is an important societal problem to solve.

* However, the major contribution of the work focusses on designing this synthetic environment, which seems to be technically significant. What would add strength to this claim would the extension of the Simulation vs Real section, expanding on how this synthetic dataset translates to real world impact There is no major algorithmic novelty introduced. Details on the reward engineering could be expanded, since that seems an essential component of the work.

* Additionally, discussion on how this work could be more generalizable to the Reinforcement Learning/Computer Vision community would be useful. There are no details if the key contributions could be adapted in other domains.  

questions: Section 4: Could details on the infrastructure to inject a Deep RL agent with the Unity environment be expanded? What is the latency of the system? Could researchers tune parameters of the simulation to allow for generalization across different scenarios?

Line 127-161: On a high level, could there be a description of the weighting of the scalar reward term? At the moment they all are considered to be equal.

   How is the upper bound of r_{t}{BC} computed in the scenario a collision takes place?

   How is wBD empirically computed? Is it a static parameter for a given experiment instance?

   It seems like a significant part of the intended behavior is being encoded through constraints.  Given the complexity of these   intricate behaviors, It would be interesting if this could be implicitly  learnt through forms of behaviour cloning, ranking relevant trajectories.





limitations: Yes
review_title: null

final_score: rating: 6: Weak Accept: Technically solid, moderate-to-high impact paper, with no major concerns with respect to evaluation, resources, reproducibility, ethical considerations.
review_content: summary: This paper proposes a multi-agent reinforcement learning algorithm to simulate the shoaling/schooling behavior of fish. The main application of the method is to control position and movement of fish in a 3D simulation which enables synthetic data generation for downstream computer vision tasks such as counting and sizing.
strengths_and_weaknesses: This paper presents an interesting practical application of reinforcement learning - simulating animal behavior for synthetic data generation.
It is great to see Deep RL applied to a practical problem! The results in the paper also seem pretty convincing: some of the rendered images from the simulation seem to closely resemble real world images.

My first point of criticism is lack of support for the claims made in the paper. Section 1 suggests that using RL for synthetic data generation in this scenario improves the accuracy of downstream vision models. The accuracy of the system is evaluated only on three videos and is never compared to anything. Typically a machine learning paper that proposes a method should compare this method's results with a baseline of some sort to support the claim that it's a good method to use for such task. Here obviously RL algorithm would be expensive to run and tune and it's advantages are not apparent unless you compare the results with obvious baselines, such as for example running Foids/Boids algorithms and comparing performance of resulting models, or even placing fish in the simulated cage randomly.
While using RL for this task definitely looks interesting, I'm not convinced that it will significantly improve the results of the final model (I would be convinced if such measurements were present in the paper).

Authors also mention that they "attempt to automate fish counting and sizing" although the sizing task is never mentioned in the paper again. I think it would have been more correct to say that this method can also be used for other task, rather than suggesting that something has been attempted but never showing the actual outcome of the attempt.

To avoid being too negative in my review, it must be said that the final results are absolutely fantastic. The supplementary video is absolutely great too. Appendix is very comprehensive and contains a lot of valuable information, especially domain expertise.
I am conflicted in my evaluation, because this is clearly a high-profile and important work addressing an important ecological problem and this paper can have a great impact in its sub-area. At the same time it lacks some of the essential elements of a machine learning publication. I think the potential impact of this paper outweights its weaknesses therefore I'm leaning towards accepting this paper.
questions: What RL implementation was used? I don't think this was mentioned in the paper.

Have the authors thought about domain randomization? I.e. randomizing textures, 3D models of fish and other parameters of data generation pipeline and how does it affect the final accuracy?
limitations: Authors briefly addressed limitations in the conclusion - from the data presented it is hard to judge the applicability of this method beyond this one task.
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: The authors present a method for augmenting computer vision task with simulation, with specific application to fish simulation. They create a simulation environment using deep reinforcement learning to mimic the behavioural properties of several fish species. The simulated behaviour is then rendered in a physically realistic computer-generated environment. Finally, computer vision modules are used downstream to, for example, identify and count fish. 
strengths_and_weaknesses: The authors provide a comprehensive and detailed simulation of fish behaviour. This is then optimized using deep reinforcement learning. They apply this approach to problems in fish farming, which they sufficiently justify in their introduction. While this work is original, with respect to the combination of many different approaches to their task problem, it is not entirely clear what is novel in the learning methods. 

In particular, the authors deal with three main problems which they address with machine/deep learning (simulation of fish behaviour, rendering fish behaviour in photo-realistic videos, identifying fish in videos). However, there is no attempt to unify learning across these problems in any significant way, so they remain relatively disjoint. Moreover, it is very difficult to judge the performance of the deep learning approach as the quantified assessment is done at the scale of fish identification, as discussed below. 

The main result of this work is the accurate simulation of fish behaviour, using deep learning. However, the presented summarising features separating out different simulation environments, swarming behaviour and change in polarity and momentum over time, are compared only against some subsets as example. These are not sufficient to judge whether this work is successful at capturing fish behaviour. I suggest the authors consider quantified metrics for the ground truth data they have on fish behaviour and systematically compare this to the simulated behaviours.
questions: Is it possible to perform some clustering on the video data and compare clustered behaviours/features to those generated using the simulation? This would give a much easier way for assessing how successful this approach has been. 

The use of simulated synthetic data to improve downstream video tasks could be a useful tool for assessing problem cases where data is sparse/collection if expensive. For example, the synthetic data could be used to identify adversarial examples where, in this example, the fish counting algorithm performs poorly. Have the authors considered this? 

Is it possible to closely couple the simulation with the available data, for example, through in-filling trajectories of fish videos given initial trajectories? However, I appreciate these agent-based simulators are highly stochastic and this may not be meaningful in practice. 
limitations: This work is very closely centred on simulating fish behaviour, aided by deep learning. The manuscript is currently structured such that it is more relevant to modelling fish behaviour than advances in machine learning. While I believe that this work represents a significant advance in reproducing the collective dynamics of fish species, the authors should consider how to structure and emphasise this work such that it has more relevance to advances in deep reinforcement learning.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The proposed paper aims at defining a fish simulation with deep reinforcement learning to mimic the behavior of swarms of fish. Fish behavior is simulated based on a multi-agent deep reinforcement learning approach and a combination of different rewards. Additionally, an underwater simulation is proposed to define plausible scenes. To validate the method, the simulation is used to generate training data for common computer vision tasks (e.g. counting of fish). 
strengths_and_weaknesses: Strengths

- Simulating the behavior of fish with deep multi-agent reinforcement learning is novel and interesting. 
- The paper is very-well written and easy to follow. 
- The approach is technically sound and the results are impressive. 
- The use of synthetic data for training neural networks for downstream tasks (counting fish) is convincing.
- The simulation is calibrated with field data. 

Weaknesses

- The only negative point I have regarding this paper is that it does not quite seem a fit for a conference paper in terms of the page limit. A lot of interesting and important content is presented in the appendix (it is 14 pages long), so it seems the paper would be a better fit for a journal. However, given the high quality of the paper I think this should not be a reason for rejecting this work. 
questions: - L113: What is meant by multi-hot values? I assume this refers to a multi-one hot vector? Please clarify. 
- L116: What exactly are the 'visual observations'? The rendered images of three frames?
- L123: 'Clamped by a small angle'? When I am not mistaken this should be 'clamped to'. 
- L152-154: While it is clear how the aggressors velocity is changed, I am missing a description of how the subordinates behavior is modeled. Put differently, what exactly is the 'chase mode' of the subordinate fish?
limitations: While Sections 5 and 6 briefly touch on the limitations of some aspects of the method (the accuracy of Yolo is actually not a limitation of the proposed method), I would encourage the authors to add a more in-depth discussion on the limitations of the proposed simulation. What are the failure cases of the simulation? What kind of fish behavior cannot be captured? etc. A discussion on the limitations could initiate future work in this direction. L314-316 also touch on a few topics but in a rather shallow manner. Here more detail would be appreciated.
review_title: null

Meta-review: metareview: This paper proposes to build photorealistic 3D models and simulation of fish schools in fish cages, with individual fish represented by swimming agents that are trained using multi-agent deep RL with bio-inspired rewards. The authors use these trained models, tuned to different fish species, to generate photorealistic images that will be fed to image YOLO-based detectors and counters, for applications to ecology and sustainable fishing. The trained fish detectors and counters is finally evaluated on real recordings.

Reviewers praised the clarity of the paper (vrph, e9JA), the motivation (vrph, ZV3m, RkxH, e9JA), the successful application of deep RL to a real problem (ZV3m, e9JA), the quality of the results (ZV3m, e9JA).

Reviewer vrph noted that the sim2real evaluation was insufficient, but the authors provided more details about comparisons of fish trajectories and distributions in the appendix. Reviewer ZV3m noted that no baseline (including Boids or random fish placement) had been used to train the fish detector and counter, and that the sizing task was mentioned but not done. Reviewer RkxH deplored the lack of metrics to qualify the learned fish behaviour. Reviewer e9JA noted that the paper was a bit long, with essential details in the appendix.

Reviewers agree on high scores (5, 6, 6, 7) and therefore I would recommend this paper for acceptance.

Thank you, Sincerely, Area Chair
confidence: Certain
Decision: Accept