DeepFoids: Adaptive Bio-Inspired Fish Simulation with Deep Reinforcement Learning
Authors: Yuko Ishiwaka
Xiao Steven Zeng
Shun Ogawa
Donovan Michael Westwater
Tadayuki Tone
Masaki Nakada
Venue: NeurIPS 2022 Accept (2022)
Keywords: Bio-inspired
Fish Schooling
Physically Based Simulation
Deep Reinforcement Learning
Adaptive

Abstract: Our goal is to synthesize realistic underwater scenes with various fish species in different fish cages, which can be utilized to train computer vision models to automate fish counting and sizing tasks. It is a challenging problem to prepare a sufficiently diverse labeled dataset of images from aquatic environments. We solve this challenge by introducing an adaptive bio-inspired fish simulation. The behavior of caged fish changes based on the species, size and number of fish, and the size and shape of the cage, among other variables. However, a method to autonomously achieve schooling behavior for caged fish did not exist. In this paper, we propose a method for achieving schooling behavior for any given combination of variables, using multi-agent deep reinforcement learning (DRL) in various fish cages in arbitrary environments. Furthermore, to visually reproduce the underwater scene in different locations and seasons, we incorporate a physically-based underwater simulation.

DeepFoids: Adaptive Bio-Inspired Fish Simulation with Deep Reinforcement Learning
Yuko Ishiwaka + Xiao S. Zeng", Shun Ogawa', Donovan Michael Westwater', Tadayuki Tone', Masaki Nakada SSoftBank Corp., Japan, 2 NeuralX Inc., USA
Abstract
Our goal is synthesize realistic underwater scenes with various fish species in different fish cages, which can be utilized to train computer vision models to automate fish counting task. It is challenging problem to prepare sufficiently diverse labeled dataset of images from aquatic environments. We solve this chal- adaptive bio-inspired fish simulation. The behavior of
lenge by introducing caged fish changes based on the species, size and number of fish, and the size and shape of the cage, among other variables. In this paper, propose method for achieving schooling behavior for any given combination of variables, using multi-agent deep reinforeement learning (DRL) in various fish cages in arbitrary environments. Furthermore, to visually reproduce the underwater scene in different locations and seasons, incorporate a physically-based underwater simulation.
Introduction
The sustainability of the food supply and the protection of marine resources two of the most important global issues, and are part of the agenda set by the United Nations to be achieved by 2030 Fish farming is one of the keys to sustainable seafood supply support the global population, as seafood consumption is growing rapidly. One of the current major problems in the fish farming industry is managing feeding. Under-feeding slows down the growth of the fish due to malnutrition. Over-feeding is even worse as it can kill the fish and the residual food contaminates the marine environment. Therefore, we attempt to automate fish counting using inexpensive commercially available RGB cameras and computer vision to optimize the amount and timing of feeding.
Deep learning has achieved great success in the field of computer vision. The quality of the dataset determines the accuracy of deep learning models, but it is difficult to obtain data to train a fish counting model. Ishiwaka et al. [1] proposed fish schooling simulation called Foids, and demonstrated the efficacy of using a CG synthetic dataset for training. Foids adds rules based on fish biology the Boids [2) algorithm. However, both Boids and Foids require manually setting a number of parameters for given fish species, which must be adjusted again should any conditions change. To solve this problem, we propose method to autonomously generate schooling behavior in fish using multi-agent deep reinforcement learning. Fish behavior in a fish farming cage depends not only on natural factors like temperature and light intensity, but also the size and shape of the fish cage, and even the species, number, and size of the fish themselves. By taking into account biological data such as the preferred light intensity, temperature, and inter-fish distance of each fish, our method achieved several distinct patterns of collective behavior depending on population density through autonomous
Yukoo Ishiwaka and Xiao S. Zeng equally contributed to this work.
iisiiikaaiiv wtw.iiiaaaag...faaa...a..a.. softbank.co co.
htsps: un, org/2030agendda
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

learning. Furthermore, we developed physically-based underwater environment simulation. This simulation capable of accurately reproducing the conditions of underwater scenes of arbitrary locations and seasons. The bio-inspired fish simulation and physically environment simulation allow for the creation of a high-quality synthetie dataset, with which we successfully trained a deep learning model to count fish of various species in fish cages.
In this paper, we explore related work in Sec. We then describe the implementation details of the simulation, data synthesis, and the computer vision models in Sec. 3 and In Sec. [ we show the
experimental results and analysis followed by a conclusion in Sec.
Related Work
It is well known that mammals use reinforcement learning
3 Experiments in neuroscience have
demonstrated that dopamine neurons play a role in rewards in reinforcement learning 4 3 (). These dopamine neurons are present not only in mammals but in flies and fish as well [7]. It has been shown that dopamine neurons are used in cooperative behavior [8] and learning [9] in fish. Therefore, we
adopted reinforcement learning for training cooperative schooling behavior.
In the field of fish swimming simulations, DRL has been used to produce efficient swimming and verify adaptation behavior [10 [1 12 [3]. These studies focused on simulations of a few fish or soft-bodied animals, but they did not cover large numbers of fish or differences between species. Tu and Terzopoulos [1] proposed a fish simulation with perception, cognition and muscle based
locomotion. Lindsey 15 defined 12 types of locomotion in fish, and Satoi al. [66 incorporated
this definition and proposed a controller which automatically selects type of locomotion based on a desired target behavior. Although those methods worked well simulating individual fish behaviors, they were not directly useful for our objective of simulating tens of thousands of fish, while forming schooling behavior organically through interactions among the fish.
DRL-based approaches have also been studied for flocking control tasks 17 simulation [20 (1). These models do not take biological specifics and their influences flocking behaviors of simulated objects into consideration, while the adaptation of these characteristics constitute a crucial part of our fish control system.
18 19
and crowd
There are relatively few publicly available computer vision datasets of underwater images. Three major such datasets OzFish (22]. Deepfish [23). and dataset from Ditria et al. 124). These datasets lack the species and environments seen in fish farms, making them unsuitable for our goals.
The biological properties we use in our simulation are aggregation, cohesion and separation rules (25 26]: fish's preferred temperature and light intensity (27 28 29]; personal space 30 3I sense of depth due to hydrostatic pressure [32 3 response to obstacles [34 35); ; and decision making interval [36), as well as the social ranking of fish. Fish often form dominance hierarchies, where the
dominant fish display aggressive behavior by chasing the subordinate individuals from time to time
37 38
The details of the biological background are described in the Appendix. In this paper, we
set rewards and penalties for DRL based on these properties.
3 Fish Simulation
In this section, we introduce a bio-inspired fish simulation trained with DRL. The bold letters v, and u are used to denote directional 3D vectors, and the uppercase calligraphic letter I denotes higher-order tensor. We structure the behavioral model fish as a multi-agent agent reinforcement learning problem, where fish agents need to navigate around constrained 3D space that resembles a fish farm environment while adapting to multiple factors that simultaneously influence their behavior. During the course of interacting with the environment and other fish, each agent acts based on its own observations and is responsible for learning a general policy that maximizes a reward. We model the policies using neural networks and optimize them using Proximal Policy Optimization (PPO) [3 with Generalized Advantage Estimation (GAE) [40].
A schematic illustration of our fish simulation pipeline is provided in Fig The parameters controlling biological and environmental factors in the fish simulation are set based on the literature presented in the Appendix and field data collected from fish farms at Onmachama and Nishiki, Japan. Fish

---

Visual Entockr
Nagtou Prelice and Viwao
Fiis State
Euteal Has eussal
128
S-8 Convy 64
Utacviice
Eerervenctal Pacars
Vieal Cisamatie Withi Sendin Bongs
Deceday Gienznnll
Cenura Dapts
Light leflet
46
Miuligenar unth Naighter Troce
Ienpensnce tilled
DEL Coured Mocils
Squel and mollion
Mirecrd Speine
Tarpst valaily
128
128
128
128
Figure 1: Simulation framework.
Figure 2: Architecture of policy network.
animation generated by the finite state machine based on the velocity of the locomotion and the fish's behavior state, which is explained in more details in the Appendix.
Biological and Environmental Factors: We incorporate important biological and environmental drivers of fish swimming patterns into the simulation framework. A fish stays within comfortable distance of its neighboring fish and aligns direction with that of nearest neighbor in the forward direction 12 41). Its activity space is constrained by the cage boundaries and water surface, which reproduce the commercial cage environment and facilitate the formation schooling. Additionally, caged fish are divided into dominant and subordinate groups, where the dominant individuals may initiate antagonistic acts by aggressively approaching the subordinate members. These factors are carefully integrated into the process of learning control policies that generate the delta velocity of fish Av! each time step / through the use of the PPO algorithm.
There are effects of underwater light intensity and temperature on the fish's vertical distribution, which we represent as A Vight and A Viemp- A fish has a preferred range of light intensity and water temperature, and it changes its vertical position to stay in the comfortable zone. The details of the computation explained in the Appendix.
In addition the bbove-mentiones components, we integrate the fish decision making interval into the framework to emulate the latency of fish's responses to environmental changes. The duration (in units of simulation steps) of the decision making interval, A Arsss for simulated species is predefined in accordance with the literature studies described in Sec. [ At the time of simulation, given the time interval of simulation steps (i)) fish updates its observation of the environment every [Arrs// Atai I steps and takes actions between updates. The delta accumulated velocity to apply at
each simulation step (Avę) can then be derived as follows:
Aviight + A Aiemp
Av Av!
(I)
Aires/ Alsim]
States and Actions: Each state 8t E state space S encodes the information fish agent observes in the environment at time step f. It can be represented by tuple (u, de, Iz), where u, is the difference between forward directions of the agent and its nearest neighboring fish in front of at current step I, de is the depth of the agent with respect to the water surface, and Z, is a visual observation tensor. During the time of exploration, fish agent collects visual observations with spatial grid sensor that imitates the sensing area of real fish [42]. The visual observation is stored as third-order tensor, whose dimensions are grid width, grid height, and number of channels. The width and height are defined by the grid resolution, which is set to be 30.20. There are 6 channels encoding a scalar value of the normalized distance from the closest detected object within the fish's 's sensing range dsense to the agent and a one-hot encoding of the object type (i.e. fish, boundary or obstacle). dyense is valued body lengths (BL) for yellowtail amberjack (yellowtail) 4 and BL. for coho salmon and seabream 42 We stack three visual observations together to infer movement before passing them to the networks. All the state components are computed the local coordinate system of the agent, with the origin located at the body center and the z-axis parallel to the fish's 's facing direction. Note

---

that a simulated fish is not capable of precisely observing its speed and rotation since real fish can only sense its relative rotation through the use of the lateral line system 44 45
The action a E action space A determined by the policy specifies the delta speed (Awy), as well as delta rotation about the x-axis (AØg) and the y-axis (Aff) in degrees. The rotation angle about the z-axis is clamped blucto small angle 03 to avoid unnatural rolling behavior. Ac, is also clamped by the maximum delta speed A Umax allowed in the cage environment. The three action components are then used to compute the delta velocity of fish Av! at the current time step and thus drive the motion of the fish agent as depicted in Eq.
Reward: The reward rt at each time step is defined encourage schooling behavior while avoiding boundary collisions and to be consistent with the biological studies described in Sec. 2
T; = =r + + + rrB +r: + + rr + r +re.
NC
(2)
The reward nf represents the penalty of colliding with the spatial boundaries, which include the cage walls and water surface. It has a fixed value of 330 if boundary collision occurs or otherwise. r:CC penalizes the collision with neighboring fish detected by box colliders using an associated weight NC and accumulates with the number of colliding agents Naa:
NC - Nuit- rt
The boundary avoidance reward r.5D encourages the fish to keep a distance from a detected spatial boundary. Its value depends on the agent's s sensing range drense- the number of detected boundaries Nond the distance di to the boundary i and the boundary avoidance weight w BD;
Niad
dhense di
BD = - ww BD t
dgense
The neighbor interaction reward ND encourages the fish to stay close to its neighbors within the sensing range and align its direction with those its neighbors. The angle A6p in degrees between the directions of the agent and each of its Nuei neighbors is calculated and a neighbor interaction weight ND used for the computation:
Nu
90 A
ND r
ND
90
On the other hand, rf penalizes energy consumption of the fish while rotating its body or adjusting speed. It is computed from a rotation penalty weight w", speed penalty weight w* together with the delta angle A0, of the body rotation and the delta accumulated speed Auf at the current time step:
=- Ad, w |
The movement reward rrM encourages the fish to swim faster than minimum speed and penalizes sudden changes in depth caused by aggressive pitch motion (around the local x-axis). the expression below, the variable led denotes the pitch angle threshold, eet is the speed threshold, 0F is the current pitch angle, and of represents the current accumulated speed:
-10 if rr < 0- < (360- gr²).
MM r
2
if frt 0 < 0r and of 2 vst.
otherwise.
Lastly, of denotes the chase reward which encourages aggression or escape behavior based on the social rank of the fish. Specifically, a dominant fish (aggressor) randomly starts a chase mode with a small probability Pa and initiates an attack on its nearest subordinate neighbor (target). This triggers the subordinate being chased to start its escape mode and swim away from the aggressor. We reward the aggressor by a fixed large value it collides with its target. This process can be expressed using the aggressor's 's accumulated velocity vf, a normalized vector d spanning from the aggressor to the target, a chase reward weight wea for the aggressor, and an escape penalty weight war for the target as follows:
RE (d. v?)
if fish an aggressor in chase mode and chases after its target.
100
if fish an aggressor in chase mode and collides with its target.
tar (d v if fish is a target in escape mode. otherwise.

---

(a) Scene
(b) 2D and 3D bounding boxes
(c) Silhouette
Figure 3: Synthetic dataset example. (a) scene rendering. (b) 2D and 3D bounding box annotations.
(c) silhouette annotation.
The chase and escape modes end for the two fish either when they collide, the target escapes to the back of the aggressor, after short period of time the dominants prefer not wasting energy prolonged chase. A detailed explanation of parameters used in action definition and reward function available in Appendix.
The simulation of realistic fish schooling behaviors be broken down into series of tasks where fish agents need to choose optimal actions from continuous action space. In the Actor-Critic framework used by PPO, a policy (actor) π is modeled that maps state s, to Gaussian distribution over action π (ae])s) The agent's 's visual observation I, which is formed as a 3D tensor, is processed by visual encoder comprising two convolutional layers and dense layer before being passed to a fully- fully-connected network with other observation components. The first and the second convolutional layers contain 16 8 filters and 32 4 filters respectively and use Leaky ReLU activation. The fully-connected layer in the visual encoder contains 128 units and also uses Leaky ReLU as the activation function. Each of the other four fully-connected layers consists of 128 units and uses the Swish activation function. The output layer is just linear layer with the output size equal the size of action space. We train the policies using PPO with a clipped surrogate objective [39]. An in-depth review of this algorithm can be found in the Appendix. The advante for the policy gradient is calculated by GAE [40]. The value function (critic) V (s;) is represented as another deep neural network with similar architecture except the output layer is single unit. The networks are built on the open source library PyTorch.
Network:
neural network (Fig. 2
4 Synthetic Dataset and Fish Counting
Our simulation is developed on Unity Engine with ML-Agents toolkit [46), and Crest Ocean System HDRP asset] is utilized to create the effect of sun and occan waves. The simulation is controlled by parameters including, but not limited to, fish biological details such as fish species, count, size, and speed; temperature and light preferences; latitude, longitude, date and time; wave size, sediment and chlorophyll concentrations; sunlight color and intensity; and light attenuation and scattering. A major trawback of prior fish simulation systems [1 1 is the need for manual parameter tuning when applying the simulation to new species new environments. However, in this work leverage the power of DRL to train the fish to adapt to changing environmental conditions given their biological data, and apply the species-specific trained models to emulate the corresponding swimming patterns. environment simulation described in the Appendix to control the
We also use the physically-basee scene with a small number of hyperparameters instead of extensive adjustment with programmable shaders. These improvements significantly automate the simulation process and allow us to generate an arbitrarily large and varied dataset by randomly setting the above mentioned parameters within physically valid range. Fig. displays an example of a synthesized dataset.
The fish counting system comprises three modules: an image pre-processing module, which converts the input video to a sequence of images and applies denoising; a fish detection module, which is a trained network based on YOLOv4 [47] with the synthetic dataset; and a fish counting module. The system takes as input a fish cage video, and outputs an estimate of the total number of fish shown in the video. The details of each component can be found in the Appendix.
availabble atlattps:/uniityya
'available at
unity, cwwc waan/paccagss/toon htw.arriiccccoooffffeettetetoet
ean- aan-system-heee hdrp- 164158 under Standard Unity Asset Store EULA license
5

---

(a) Frames: real video
(b) Frames: simulation.
Figure 4: Comparison of the scenes captured from different views a fish farm and in the simulation. Left-to-right: yellowtail, red seabream and coho salmon. Note that a fish-eye lens is used for yellowtail while a regular lens is used for coho salmon and red seabream.
5 Experiments and Results
We describe the training process, experimental results of the fish behavioral model with diverse biological and cage settings, underwater simulation, and fish counting.
Training: We trained the behavioral models of coho salmon, yellowtail and red seabream. A two-step training scheme was employed to facilitate the adaption for agents in eleven environments with the same action space, observation space and types of detectable objects, but different fish species, fish sizes, fish numbers, cage sizes and cage shapes. We pretrained the three species in their corresponding default scenes, which contain 1000 coho salmon, 45 yellowtail and 10 red seabream, respectively, with the cage configurations set to be the same as the fish farm environments we collected field data from. The pretraining results were usually sufficient to enable the agents to navigate and avoid cage boundaries to some extent, but collisions between fish and with boundaries still occur and unnatural movements persist. We then performed transfer learning based on the pretrained policies for each species in its corresponding environment-spectn the default scene that is used to reproduce the real scene. We found the use of such two-step training scheme led to faster convergence and overall performance improvements compared to training from scratch for each environment. We enabled early termination of episode if the agent collides with either a cage wall or the water surface. The agent would subsequently restart a new episode from a random position within the cage with random valid rotation and an initial speed vo- This strategy discourages boundary collision and prevents the agents from collecting data that is irrelevant to boundary avoidance. The values of hyperparameters in the pretraining and transfer learning phases are similar if not identical. For yellowtail and red seabream, we ran a total number of two million time steps and used linearly decaying learning rate of 0.001 in both phases. For coho salmon, adopted the same settings, but increased the pretraining and transfer learning steps to four million and one million time steps respectively, because the 1000 coho salmon in the default scene is much more than the other two species. The total wall-clock training times for four million and five million simulation steps were around 5500 seconds and 6800 seconds, respectively. A detailed explanation of the eleven simulation environments. hyperparameter settings, and the learning curves of two training stages are available the Appendix. We also demonstrate the training progress of yellowtail in a recorded demo that is part of our supplementary video. The training took place on an NVIDIA GeForce RTX 3080 Laptop GPU installed on Windows 10 machine with 3.3 GHz AMD Ryzen 5900HX CPU.
experiments, including
We simulated swimming scenes of the above-mentioned three species in order to provide a variety of synthetic data to train the computer vision system that detects and tracks fish. Fish agents were initialized at random positions in the cage with random valid rotations, and would start to form circling patterns by following the trained policies. A qualitative comparison between the image sequences from the simulation and the videos captured underwater is displayed in Fig. Note that the swimming patterns of the three fish species are distinct in the captured video. Our simulation successfully reproduced such behavioral differences as well as the environmental variation such as lighting, water color and turbidity.
Fish Behavior:
We also showcase the simulation results of five example environments with different species, fish size, fish number, cage size and shape settings in Fig. 5. A body scale of 1.0 refers to the default body lengths (BI.) of coho salmon, yellowtail and red seabream, which were set to 0.49 meters, 0.52 meters, and 0.34 meters respectively according to the collected field data. We additionally applied

---

mild repulsive force acting as an tojeet-to-object colliding force to push any penetrating fish away from each other to further facilitate collision avoidance among agents. The fish swimming patterns vary with the scene configuration since they result from combination of factors discussed in Sec 3 Notably, the 10 red seabream in Fig. 5 a: left 3) slowly circle around the cage center given the small cage size, whereas the same 10 fish in Fig. 5 a| right 3) learn to swim rapidly and away from the center since the cage size is much larger. On the other hand, the red seabream in Fig. (5)) form a more compact school and slowly circle around the entire small cage because of their larger quantity. In addition, we simulated two fictional scenes where fish with large body size disparity (Fig- F(c)) and multiple species (Fig. ((d) are mixed together. With associated biological measurements and trained policies, fish in each of the size and species groups exhibit distinct swimming patterns while influencing those of other groups in the same fish cage.
Simulation vs Reality: We focus on the relationship between fish density in a cage and the resulting fish school states, and compare simulation results with data obtained in the field. We use two kinds of schools in our simulations, sparse (0.59 fish/ and dense (14.7 fish/m' hmn,), where the densities are set to be consistent with the field work. In the real fish farms, the sparse cage is 6.5m x 6.5m square with a depth of 6m and contains 272 fish. The dense cage is an octagon with 6.5m edges and a depth of 10m and contains about 30000 fish. We quantify states of schools, and focus on milling, characterized by regular circulation around an axis, and swarming, in which fish aggregate with a weak polarization [48 49). Fig 6 shows sampled trajectories, polar order parameter P given by the norm of the average of direction vectors for each fish in a school, and a normalized angular momentum M representing how regular the circulation is [49] for two schools with different densities. Details can be found in the Appendix. The simulation result demonstrates that the collective behavior of the school transitions from swarming to milling as density increases, and this tendency is in good agreement with the real data. Several trajectories of individual fish in real video in both sparse and dense schools are exhibited in Fig. 7 and they resemble the ones in simulation results drawn by the bold trajectories in the right panels.
We initialized the scene with 20% of fish being dominant and the rest being subordinate based on the reported distribution of social hierarchies among fish in (38). The dominant individuals would occasionally chase their subordinate neighbor until colliding with or losing sight of the latter, which matches our observation of the fish behavior in underwater videos captured in aquaculture sites. A demonstration of the simulation scenes is also included in our supplementary video.
Moreover, we show the influence of the dominance hierarchy on fish behavior in Fig. g
We compare the results of simulations trained by PPO and SAC [50], in the default environment of coho salmon, in which 1000 coho salmon swim in a large octagonal cage with edges of 3 meters and a height of 4.6 meters. Both PPO and SAC eventually produced agents forming swimming patterns that resembled those of real coho salmon in reference videos, while the former yielded more stable outcomes. The same reward parameter setting used for PPO was initially applied to SAC in the pretraining stage. Although the SAC agents collected a higher average reward over episodes from an early stage of training than their PPO counterparts, they became over-trained in the later phase of training and exhibited a circling pattern where all salmon tended to cruise in the same direction and align their body rotations, which would be unnatural in real-world farming cage setting. This result was jointly caused by a higher sample efficiency and entropy maximization strategy of the SAC algorithm that encourages exploration. After adjusting the reward parameters accordingly, SAC agents were able to swim in equally natural patterns as PPO agents after the pretraining and fine tuning stages. However, no significant improvement in the cumulative average reward was observed in the fine tuning stage of SAC and its resulting value was lower than that of PPO as shown in Fig. 9 A possible explanation is that the aggregated penalties from constraining factors such as boundary and neighbor collisions overwhelmed SAC agents as their off pooicy nature required them collect experience from previous episodes into the replay buffer for future updates.
PPO vs SAC:
Underwater Simulation: The results of the experiments with each component of the environment
simulation are illustrated in Fig. 10 Fig. [T]aa shows how the underwater scene becomes more reddish brown with higher concentrations of chlorophyll resulting in an increasing attenuation of blue. A higher value of sediment concentration makes the water cloudier as shown in Fig. [o(b. Fig. [0] c) depicts a timelapse of the scene. Fig. d) demonstrates the change in brightness with increasing depth, since more light is absorbed at greater depths. Fig. 10 shows the influence of changing
7

---

(a) 10 red scabream in cubic cage with edge length of 3 meters (left three) and meters (right three).
(b) 45 red seabream in a cubic cage with an edge length of 3 meters.
1000 coho salmon with body scales 0.5 (yellow), (blue) and 1.5 (red) an octagon cage with edge length of 3 meters. The simulated fish form two vertical layers based on their body sizes.
100 coho salmon (blue), 100 yellowtail (yellow) and 100 red seabream (red) in octagon cage with an edge length of meters. swimming pattern vary with their distinct biological factors.
Figure 5: Sequence of frames from example environments with diverse fish simulation configurations. All fish have random body scales within the interval [0.9, 1.1] by default unless specified.
1.0
Sparse
Dense
05
2.05
Sparse
Dense
200
(a) Sampled trajectories
(b) Charactoristics school
Figure 6: Relation between density and states (swarming milling) of fish schools and their characteristics: (a) 5 trajectories in sparse (0.59 fish/m² left) and dense (14.7 fish/m' right) schools. The left and right panels show swarming and milling state respectively. The fish cages square with 4.0m edges in the left panel and octagon with 1.66m edges the right panel. (b) Time series of P (left) and M (right) for dense (blue line) and sparse (red dashed line) schools.
small particle concentration on the brightness of the scene. A higher concentration of small particles increases the amount of light scattered, both towards and away from the camera, causing light around the sun to brighten and everything away from the light source to darken. Fig. [0] shows the influence large particle concentration, which works the same way as small particle concentration but has smaller effeet light scattering.

---

(a) Sparse school
(b) Dense school
Figure 7: Fish trajectories in schools in real video (left panels) of coho salmons and simulated ones (right panels). The number of fish and the size and shape of the fish cage in each simulation are described the caption in Fig. 6 The parameters of real ones are as follows (a) sparse school (the fish count is 615) in rectangular cage with 6.5m edge and 6m depth. Some fluctuations are observed on the orbits in real videos because the camera cannot be fixated completely the ocean. (b) dense school (the fish count is approximately 30000) in octagon cage with 6.5m edge and 10m depth.
(a) Simulation results.
(b) Real recordings.
Figure 8: Sequence frames from an aggressive behavior instance where a dominant fish (with red
trajectory) chases after subordinate neighbor (with yellow trajectory) until hitting the latter.
Fish Counting: We applied our modified YOLOv4 model 47 trained with the synthetic dataset real footage. Fig. [] shows some results with the videos of coho salmon, yellowtail and red seabream. The trained fish counting algorithm estimated the number of fish in each frame, and the total number of fish was computed by taking the max of the estimated values while the network processes all the video frames. The counting results for coho salmons, yellowtails and red seabreams are 69:61, 22:32, and 7:7 (leftrour system, right: manually counting) respectively, When the density of the fish is high and many occlusions are observed, the accuracy of the counting system drops.
6 Conclusions and Future Work
We presented bio-inspired fish simulation system with DRL which adapts to various conditions autonomously by applying transfer learning, and which enabled us to simulate a broad range of fish behaviors in many scenarios. In addition, introduced a physically-based underwater simulation which enabled us to simulate various undersea conditions. Using these, we were able to synthesize large dataset with accurate annotations which we could use for computer vision tasks. We presented one application, fish counting, in this paper, but the potential of this synthetic dataset created with our novel simulations goes beyond fish counting. We will extend the application to 3D detection, pixel-wise segmentation, size estimation and tracking tasks in order to make more useful tools for fish farms.
limitation of the current pipeline is the manual setting of weights for reward functions. We will explore techniques that dynamically adjust weights based on the performance of the DRL controller in the training process. Moreover, we chose not to apply any behavior cloning method because it was difficult collect and annotate enough samples of fish trajectory data from real video. which was the main challenge we aimed to solve with the proposed method for the preparation of the dataset for computer vision models. However, the fine-tuning of the trained DRL model using behavior cloning with small sample of the real fish behavioral data is an interesting method to enhance the reality of the simulation that can be worth working on.
"uunder Apache License 2.0

---

Coho Salmon (Pretrain) PPO vs SAC
Coho Salmon (Fine Tuning) PPO vs SAC
PPO SAC
9k-
PPO SAC
6000
Mean reward
Mean reward
3000
6k-
3k-
-3000
-5000
1M
2M
3M
0.2M
0:4M
0.6M1
0.8M
1M
Time steps
Time steps
Figure 9: Comparison of per-episode mean cumulative rewards over all coho salmon agents trained using PPO (blue) and SAC (red) algorithms in pretraining (left) and fine tuning (right) phases.
(a) Chlorophyll concentration mg/m².
(b) Sediments concentration from 0 g/m
(c) Time-lapse from noon to pm.
(d) Light attenuation from 0 10 meters depth.
(c) Light scattering with small particles (<1 micron in (f) Light scattering with large particles (> micron in diameter) concentrations from 0 ppm to 4 ppm. diameter) concentrations from 0 ppm to 4 ppm.
Figure 10: Experiments with each component of the environment simulation.
In this paper, excluded the effect of tidal currents and the influence of fish body temperature movement dynamics from our simulations, and the proposed simulation method is not applicable to special occasions such as fish behavior in feeding and disease conditions yet. We plan to incorporate them to make our simulation more biologically accurate.
Acknowledgment
Suda, T. Yoshida, G. Yasui and others from SoftBank Corp., Mizutani, Tsurumi, S. Sone, and others from Nosan Corporation for field works. We also thank M. L. Eastman and S. Kakazu from SoftBank Corp. for their supports on computer graphics pipeline and M. A. N. Kemas from NeuralX Inc. for computer vision models.
We are grateful
Figure 11: Example images of fish counting system using YOLOv4 trained model. The captured video data with (left) coho salmon (middle) yellowtail (right) red seabream were fed to our fish counting pipeline, and the total number of fish in each cage was estimated automatically.
10

---

References
[1] Yuko Ishiwaka, Xiao S Zeng, Michael Lee Eastman, Sho Kakazu, Sarah Gross, Ryosuke Mizutani, and Masaki Nakada. Foids: bio-inspired fish simulation for generating synthetic datasets. ACM Transactions on Graphics (TOG), 40(6):1-15, 2021.
[2] Craig W Reynolds. Flocks, herds and schools: A distributed behavioral model. pages 25-34, 1987.
[3] Burrhus Frederic Skinner. A case history in scientific method. American psychologisi, 11(5):221, 1956.
[4] Wolfram Schultz, Peter Dayan, and P. Read Montague. A neural substrate of prediction and reward. Science, 275(5306) 1997.
[5] Paul W Glimcher. prediction error hypothesis. Proceedings of the National Academy of Sciences, 108(Supplement 31:15647-15654, 2011.
Understanding dopamine and reinforcement learning: the dopamine reward
[6] P Read Montague and Gregory S Berns. Neural economics and the biological substrates of valuation. Neuron, 36(2):265-284, 2002.
[7] Fangmiao Sun, Jianzhi Zeng. Miao Jing, Jingheng Zhou, Jiesi Feng, Scott Owen, Yichen Luo, Funing Li, Huan Wang, Takashi Yamaguchi, et al. A genetically encoded fluorescent sensor enables rapid and specific detection of dopamine in flies, fish, and mice. Cell, 174(2):481-49 2018.
[8] João PM Messias, José R Paula, Alexandra S Grutter, Redouan Bshary, and Marta Soares. Dopamine disruption increases negotiation for cooperative interactions in a fish. Scientific reports, 6(1):188, 2016.
[9] Joao PM Messias, Teresa Santos, Maria Pinto, and Marta Soares. Stimulation of dopamine dl receptor improves learning capacity in cooperating cleaner fish. Proceedings of the Royal Society B: Biological Sciences, (83((223)20152222,, 2016.
[10] Siddhartha Verma, Guido Novati, and Petros Koumoutsakos. Efficient collective swimming by hamnsssing vortices through deep reinforcement learning. Proceedings of the National Academy of Sciences, 115(23):5849-583 2018
[11] Yi Zhu, Fang-Bec Bao Tian, John Young, James C Liao, and Joseph CS Lai. A numerical study of fish adaption behaviors complex environments with a deep reinforcement learning and immersed boundary-latiice boltzmann method. Scientific Reports, 11(1):1-20, 2021.
[12] Misha B Ahrens. Zebrafish neuroscience: Using artificial neural networks to help understand brains. Current Biology, 29(21): 100-:R11188RR111 2019.
[13] Sehee Min, Jungdam Won, Seunghwan Lee, Jungnam Park, and Jehee Lee. Softcon: Simulation and control of soft-bodied animals with biomimetic actuators. ACM Transactions on Graphics (TOG), 38(6):1-12, 2019.
[14] Xiaoyuan Tu and Demetri Terzopoulos. Artificial fishes: Physics, locomotion, perception, behavior. page 43-50, 1994.
[15] Casimir Charles Lindsey. form, function, and locomotory habits in fish. Fish Physiology. 7:1-100, 1978.
[16] Daiki Satoi, Mikihiro Hagiwara, Akira Uemoto, Hisanao Nakadai, and Junichi Hoshino. Unified motion planner for fishes with various swimming styles. ACM Transactions on Graphics (Proc. SIGGRAPH). 35(4), 2016.
[17] Peter Sunehag, Guy Lever, Siqi Liu, Josh Merel, Nicolas Heess, Joel Z Leibo, Edward Hughes, Tom Eccles, and Thore Graepel. Reinforcement learning agents acquire flocking and symbiotic behaviour simulated ecosy stems. ALIFE 2019: The 2019 Conference on Artificial Life, pages 103-110. 10. MIT Press, 2019.

---

[18] Pengming Zhu, Wei Dai, Weijia Yao, Junchong Ma, Zhiwen Zeng, and Huimin Lu. Multi-robot flocking control based on deep reinforeement learning. IEEE Access, 8:150397-150406, 2020.
[19] Huiyang Yu, Bo Liu, Chengyun Wang, Xuechao Liu, Xi- Yun Lu, and Haibo Huang. Deep- Sinnforemeentleannnng--eee self-organization of freely undulatory swimmers. Phys. Rev. E, 105:045105, Apr 2022.
[20] Jaedong Lee, Jungdam Won, and Jehee Lee. Crowd simulation by deep reinforcement learning. In Proceedinggs of the I 11th Annual International Conference on Motion, Interaction, and Games, MIG 18, New York, NY, USA, 2018. Association for Computing Machinery.
[21] Qingging Wang, Hong Liu, Kaizhou Gao, and Le Zhang. Improved multi reinforcement learning for path planning-baue based crowd simulation. IEEE Access, 7:73841-73855 2019.
[22] University of Western Australia (UWA) Australian Institute of Marine Science (AIMS) and Curtin University. Ozfish dataset- machine learning dataset for baited remote underwater video stations. 2019.
[23] Alzayat Salch, Issam H. Laradji, Dmitry A. Konovalov, Michael Bradley, David Vazquez, and Marcus Sheaves. A realistic fish-habitat dataset to evaluate algorithms for underwater visual analysis. Scientific Reports, 10(1), Sep 2020.
[24] Ellen M Ditria, Rod M Connolly, Eric L. Jinks, and Sehastian Lopez-Marcano. Annotated video footage for automated identification and counting of fish in unconstrained marine environments, 2021.
[25] Charles M. Breder and Florence Halpern. Innate and acquired behavior affecting the aggregation of fishes. Physiological Zoology, 19(2)1154-190, 1946.
[26] William Wallace Reynolds and Martha Elizabeth Casterlin. Behavioral thermorcgulation and the "final preferendum" paradigm. American zoologist, 19(1):111-224, 1979.
[27] Arne Jensen, Bjørn O Johnsen, and Laila Saksgård. Temperature requirements in atlantic salmon (salmo salar), brown trout (salmo trutta), and arctic char (salvelinus alpinus) from hatching to initial feeding compared with geographic distribution. Canadian Journal of Fisheries and Aquatic Sciences, 46(5):786-789, 1989.
[28] I Huse and JC Holm. Vertical distribution of atlantic salmon (salmo salar) as a function of illumination. Journal of Fish Biology, 43:147-156, 1993.
[29] Anders Fernö, Ingvar Huse, Jon-Erik Juell, and Asmund Bjordal. Vertical distribution of atlantic salmon (salmo solar 1.) in net pens: trade-off between surface light avoidance and food attraction. Aquaculture, 132(3-4):285-290) 1995.
[30] lon-Erik Juell. The behaviour of atlantic salmon in relation to efficient cage-rearing. Reviews in fish biology and fisheries. 5(3):320-335 1995.
[31] Martin Føre, Tim Dempster, Jo Arve Alfredsen, Vegar Johansen, and David Johansson. Mod- elling of atlantic salmon (salmo salar 1.) behaviour in sea-cages: lagrangian approach. Aquaculture, (98(3-4):196221 2009.
[32] G. Macaulay, Wright, F. Oppedal, and T. Dempster. Buoyancy matters: Establishing the maximum neutral buoyancy depth of atlantic salmon. Aquaculture, 519:734925, 2020.
[33] Victoria A Davis, Robert I Holbrook, and Theresa Burt de Perera. Fish can use hydrostatic pressure to determine their absolute depth. Communications biology, 4(1): 2021.
[34] Lucy Odling-Smee and Victoria Braithwaite, The role of learning in fish orientation. Fish and Fisheries, 4(3):235-244 2003.
[35] Alisha Brown, Marcia Spetch, and Peter I. Hurd. Growing circles: Rearing environment alters spatial navigation in fish. Psychological Science, 18(7):569-573, 2007.
12

---

[36] Thomas H. Miller, Katie Clements, Sungwoo Ahn, Choongseok Park, Eoon Hye Ji, and Fadi A. Issa. Social status-dependen shift in neural circuit activation affects decision making. Journal of Neuroscience, 37(8):2137-2148 2017.
[37] Chiweyite Ejike and Carl B Schreck. Stress and social hierarchy rank in coho salmon. Transac- tions of the American Fisheries Society, 109(4):423 426, 1980.
[38] Yoshitaka Sakakura and Katsumi Tsukamoto. Social rank in schools of juvenile yellowtail, seriola quinqueradiata. Journal of Applied Ichthyology, 14(1-2):69-73, 1998.
[39] John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford. and Oleg Klimov. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06:: 2017.
[40] John Schulman, Philipp Moritz, Sergey Levine, Michael Jordan, and Pieter Abbeel. High- dimensional continuous control using generalized advantage estimation. arXiv preprint arXiv: 150.023500.022338. 2015.
[41] James E Herbert-Rean Read, Andrea Perna, Richard P Mann, Timothy M Schaerf, David JT Sumpter, and Ashley JW Ward. Inferring the rules of interaction of shoaling fish. Proceedings of the National Academy of Sciences, 108(46):18766-17711 2011.
[42] Andreas Huth and Christian Wissel. The simulation of the movement of fish schools. Journal of theoretical biology, 156(3):365-385, 1992.
[43] Yoshitaka Sakakura and Katsumi Tsukamoto. Effects of density, starvation and size difference on aggressive behaviour in juvenile yellowtails (seriola quinquevadiata). Journal of Applied Ichthyology, (4(1-2):9-13, 13, 1998.
[44] Yonggang Jiang, Zhiqiang Ma, and Deyuan Zhang. Flow field perception based on the fish lateral line system. Bioinspiration biomimetics, 14(4):041001, 2019.
[45] Leif Ristroph, James C Liao, and Jun Zhang. Lateral line layout correlates with the differential hydrodynamic pressure swimming fish. Physical Review Letters, 114(1):01810,, 2015.
[46] Arthur Juliani, Vincent-Pierre Berges, Ervin Teng, Andrew Cohen, Jonathan Harper, Chris Elion, Chris Goy, Yuan Gao, Hunter Henry, Marwan Mattar, et al. Unity: A general platform for intelligent agents. arXiv preprint /
2018.
[47] Alexey Bochkovskiy, Chien- Yao Wang, and Hong- Yuan Mark Liao. Yolov4: Optimal speed and accuracy of object detection. arXiv preprint arXiv:2004.1199 2020.
[48] Johann Delcourt and Pascal Poncin. Shoals and schools: back to the heuristic definitions and quantitative references. Reviews in Fish Biology and Fisheries, 22(3):595-619, 2012.
[49] Daniel S Calovi, Ugo Lopez, Sandrine Ngo, Clément Sire, Hugues Chaté, and Guy Theraulaz. Swarming, schooling, milling: phase diagram of data-driven fish school model. New Journal of Physics, 16(1):015026, jan 2014.
[50] Tuomas Haarnoja, Aurick Zhou, Pieter Abbeel, and Sergey Levine. Soft actor-critic: Off- policy maximum entropy deep reinforcement learning with a stochastic actor. In International conference on machine learning, pages 1861-1870. PMLR, 2018
13


Reviews:
[{"review_id": "tQQVPSTDyNu", "reviewer": "Reviewer_RkxH", "summary": "The authors present a method for augmenting computer vision task with simulation, with specific application to fish simulation. They create a simulation environment using deep reinforcement learning to mimic the behavioural properties of several fish species. The simulated behaviour is then rendered in a physically realistic computer-generated environment. Finally, computer vision modules are used downstream to, for example, identify and count fish. ", "questions": "Is it possible to perform some clustering on the video data and compare clustered behaviours/features to those generated using the simulation? This would give a much easier way for assessing how successful this approach has been. \n\nThe use of simulated synthetic data to improve downstream video tasks could be a useful tool for assessing problem cases where data is sparse/collection if expensive. For example, the synthetic data could be used to identify adversarial examples where, in this example, the fish counting algorithm performs poorly. Have the authors considered this? \n\nIs it possible to closely couple the simulation with the available data, for example, through in-filling trajectories of fish videos given initial trajectories? However, I appreciate these agent-based simulators are highly stochastic and this may not be meaningful in practice. ", "limitations": "This work is very closely centred on simulating fish behaviour, aided by deep learning. The manuscript is currently structured such that it is more relevant to modelling fish behaviour than advances in machine learning. While I believe that this work represents a significant advance in reproducing the collective dynamics of fish species, the authors should consider how to structure and emphasise this work such that it has more relevance to advances in deep reinforcement learning.  \n\n\n", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The authors provide a comprehensive and detailed simulation of fish behaviour. This is then optimized using deep reinforcement learning. They apply this approach to problems in fish farming, which they sufficiently justify in their introduction. While this work is original, with respect to the combination of many different approaches to their task problem, it is not entirely clear what is novel in the learning methods. \n\nIn particular, the authors deal with three main problems which they address with machine/deep learning (simulation of fish behaviour, rendering fish behaviour in photo-realistic videos, identifying fish in videos). However, there is no attempt to unify learning across these problems in any significant way, so they remain relatively disjoint. Moreover, it is very difficult to judge the performance of the deep learning approach as the quantified assessment is done at the scale of fish identification, as discussed below. \n\nThe main result of this work is the accurate simulation of fish behaviour, using deep learning. However, the presented summarising features separating out different simulation environments, swarming behaviour and change in polarity and momentum over time, are compared only against some subsets as example. These are not sufficient to judge whether this work is successful at capturing fish behaviour. I suggest the authors consider quantified metrics for the ground truth data they have on fish behaviour and systematically compare this to the simulated behaviours.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "sUNaOKASBR3", "reviewer": "Reviewer_ZV3m", "summary": "This paper proposes a multi-agent reinforcement learning algorithm to simulate the shoaling/schooling behavior of fish. The main application of the method is to control position and movement of fish in a 3D simulation which enables synthetic data generation for downstream computer vision tasks such as counting and sizing.", "questions": "What RL implementation was used? I don't think this was mentioned in the paper.\n\nHave the authors thought about domain randomization? I.e. randomizing textures, 3D models of fish and other parameters of data generation pipeline and how does it affect the final accuracy?", "limitations": "Authors briefly addressed limitations in the conclusion - from the data presented it is hard to judge the applicability of this method beyond this one task. \n", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "This paper presents an interesting practical application of reinforcement learning - simulating animal behavior for synthetic data generation.\nIt is great to see Deep RL applied to a practical problem! The results in the paper also seem pretty convincing: some of the rendered images from the simulation seem to closely resemble real world images.\n\nMy first point of criticism is lack of support for the claims made in the paper. Section 1 suggests that using RL for synthetic data generation in this scenario improves the accuracy of downstream vision models. The accuracy of the system is evaluated only on three videos and is never compared to anything. Typically a machine learning paper that proposes a method should compare this method's results with a baseline of some sort to support the claim that it's a good method to use for such task. Here obviously RL algorithm would be expensive to run and tune and it's advantages are not apparent unless you compare the results with obvious baselines, such as for example running Foids/Boids algorithms and comparing performance of resulting models, or even placing fish in the simulated cage randomly.\nWhile using RL for this task definitely looks interesting, I'm not convinced that it will significantly improve the results of the final model (I would be convinced if such measurements were present in the paper).\n\nAuthors also mention that they \"attempt to automate fish counting and sizing\" although the sizing task is never mentioned in the paper again. I think it would have been more correct to say that this method can also be used for other task, rather than suggesting that something has been attempted but never showing the actual outcome of the attempt.\n\nTo avoid being too negative in my review, it must be said that the final results are absolutely fantastic. The supplementary video is absolutely great too. Appendix is very comprehensive and contains a lot of valuable information, especially domain expertise.\nI am conflicted in my evaluation, because this is clearly a high-profile and important work addressing an important ecological problem and this paper can have a great impact in its sub-area. At the same time it lacks some of the essential elements of a machine learning publication. I think the potential impact of this paper outweights its weaknesses therefore I'm leaning towards accepting this paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "PQmvPT0eV2", "reviewer": "Reviewer_vrph", "summary": "The given work develops a fish simulation motivated by solving real world problems in the fish farming industry such as over and underfeeding. Through this work, they wish to replicate fish schooling behaviour with higher fidelity, by leveraging Deep Reinforcement Learning to model this. The authors argue that this resulting setup could then be used for training computer vision models to accurately count varied fish species in the cages.", "questions": "Section 4: Could details on the infrastructure to inject a Deep RL agent with the Unity environment be expanded? What is the latency of the system? Could researchers tune parameters of the simulation to allow for generalization across different scenarios?\n\nLine 127-161: On a high level, could there be a description of the weighting of the scalar reward term? At the moment they all are considered to be equal.\n\n   How is the upper bound of r_{t}{BC} computed in the scenario a collision takes place?\n\n   How is wBD empirically computed? Is it a static parameter for a given experiment instance?\n\n   It seems like a significant part of the intended behavior is being encoded through constraints.  Given the complexity of these   intricate behaviors, It would be interesting if this could be implicitly  learnt through forms of behaviour cloning, ranking relevant trajectories.\n\n\n\n\n", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "* Overall, the work is well written, well organized and clear to follow. It reasonably describes the past work in the field, stating how the contributions are distinct\n\n* The authors motivate the problem statement well, on how this is an important societal problem to solve.\n\n* However, the major contribution of the work focusses on designing this synthetic environment, which seems to be technically significant. What would add strength to this claim would the extension of the Simulation vs Real section, expanding on how this synthetic dataset translates to real world impact There is no major algorithmic novelty introduced. Details on the reward engineering could be expanded, since that seems an essential component of the work.\n\n* Additionally, discussion on how this work could be more generalizable to the Reinforcement Learning/Computer Vision community would be useful. There are no details if the key contributions could be adapted in other domains.  \n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "8H4AkDMnhc-", "reviewer": "Reviewer_e9JA", "summary": "The proposed paper aims at defining a fish simulation with deep reinforcement learning to mimic the behavior of swarms of fish. Fish behavior is simulated based on a multi-agent deep reinforcement learning approach and a combination of different rewards. Additionally, an underwater simulation is proposed to define plausible scenes. To validate the method, the simulation is used to generate training data for common computer vision tasks (e.g. counting of fish). ", "questions": "- L113: What is meant by multi-hot values? I assume this refers to a multi-one hot vector? Please clarify. \n- L116: What exactly are the 'visual observations'? The rendered images of three frames?\n- L123: 'Clamped by a small angle'? When I am not mistaken this should be 'clamped to'. \n- L152-154: While it is clear how the aggressors velocity is changed, I am missing a description of how the subordinates behavior is modeled. Put differently, what exactly is the 'chase mode' of the subordinate fish?", "limitations": "While Sections 5 and 6 briefly touch on the limitations of some aspects of the method (the accuracy of Yolo is actually not a limitation of the proposed method), I would encourage the authors to add a more in-depth discussion on the limitations of the proposed simulation. What are the failure cases of the simulation? What kind of fish behavior cannot be captured? etc. A discussion on the limitations could initiate future work in this direction. L314-316 also touch on a few topics but in a rather shallow manner. Here more detail would be appreciated.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths\n\n- Simulating the behavior of fish with deep multi-agent reinforcement learning is novel and interesting. \n- The paper is very-well written and easy to follow. \n- The approach is technically sound and the results are impressive. \n- The use of synthetic data for training neural networks for downstream tasks (counting fish) is convincing.\n- The simulation is calibrated with field data. \n\nWeaknesses\n\n- The only negative point I have regarding this paper is that it does not quite seem a fit for a conference paper in terms of the page limit. A lot of interesting and important content is presented in the appendix (it is 14 pages long), so it seems the paper would be a better fit for a journal. However, given the high quality of the paper I think this should not be a reason for rejecting this work. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept