# Tracking Full-Body Motion of Multiple Fish with Midline Subspace Constrained Multicue Optimization

Authors: Xiang Liu; Pei Ru Zhu; Ye Liu; Jing Wen Zhao
Journal: Scientific Programming
 (2019)

---

## Abstract

Capturing the body motion of fish has been gaining considerable attention from scientists of various fields. In this paper, we propose a method which is able to track the full-body motion of multiple fish with frequent interactions. We firstly propose to model the midline subspace of a fish body which gives a compact low-dimensional representation of the complex shape and motion. Then we propose a particle swarm-based optimization framework whose objective function takes into account multiple sources of information. The proposed multicue objective function is able to describe the details of fish appearance and is also effective through mutual occlusions. Excessive experimental results have demonstrated the effectiveness and robustness of the proposed method.


---

## Body

## 1. Introduction

The most effective way to quantitatively research behavior patterns and underlying rules of fish schools is tracking each fish. It is also helpful in many related applications such as robotics and virtual reality. For example, based on the body motion of fish, people can build man-made fish-like swimming robots and create vivid virtual fish in computers [1–9]. Top-view video and 2D tracking obtain sufficiently informative motion data for behavior investigation because a shallow water tank is used in many fish behavior research experiment and the fish typically swim around the same horizontal plane. However, tracking the full-body motion of fish typically multiple fish with interactions is still a challenging task due to (1) full-body motion of fish is highly complex which is difficult to model with a few parameters. (2) Fish may move abruptly thus the motion continuity assumption no longer hold, which causes conventional tracking approaches to fail. (3) Multiple fish cause frequent mutual occlusions which corrupt the appearance models of the tracking approaches.Visual tracking is a hot research topic during the past two decades, and significant improvements have been made in all aspects of visual tracking such as appearance model [10, 11, 12] and estimation method [13, 14]. And multiple trackers can be utilized for tracking multiple targets [15–17]. Nevertheless, conventional visual trackers which are designed for tracking the positions of generic objects are not applicable to the full-body motion tracking problem here. Another branch of multitarget tracking methods follows the detection and association framework [18, 19], i.e., the outputs of detectors are associated with trackers across time. In this problem, however, the detectors may fail to give correct output during mutual occlusions which frequently happen and last for sufficiently long time, causing difficulties for the subsequent data association.To be truly helpful for biological research, many numbers of automatic software were developed for multiobject tracking, such as ANY-maze and EthoVision [2, 20–22]. But only a few targets can be tracked and professional experiment setup are needed. A multiple tracking system of fish on the basis of a scale-space determinant of Hessian (DoH) fish head detector and Kalman filter is developed by Qian et al. [23]. Delcourt et al.’s system can track as many as 100 fish simultaneously but is not suitable for long period tracking [2]. However, these approaches highly depend on detection results and motion continuity for data association, and discriminative information of the head is not fully exploited. The body modeling and motion tracking of multiple fish is preliminarily discussed and attempted [24]; however, the effect declines when the number of fish increases.Problems of tracking will be more difficult when severe occlusions occur: individuals may be assigned wrong identities and these errors would propagate throughout the rest of the video. Several existing tracking methods combine detection and tracking stages together to correct detection errors timely. Prior knowledge on possible articulations and temporal coherency is used by Andriluka et al. to associate the detection of each individual across frames, which depends on the motion model and object specific codebooks constructed by clustering local features. Kalal et al. [10] proposed the framework tracking-learning-detection (TLD) to track single object, which is divided into tracking, learning, and detection 3 subtasks. The tracker follows the target across frames; the detector localizes the object in each frame and is corrected, and it is updated online by P-N learning. Wang et al. [25] proposed an effective tracking method using convolutional neural network (CNN) for head identification. They firstly detect fish heads using a scale-space method, and data association across frames is then achieved via identifying the head image pattern of each individual fish in each frame via CNN specially tailored to suit this task. Finally they combine prediction of the motion state and the recognition result by CNN to associate detection across frames. But samples must be collected for training CNN, and when new targets occur in the video, their method cannot work.We present in this paper a method that is capable of tracking the full-body deformation of multiple fish with frequent mutual interactions. Since in most of the time, fish motion and deformation are horizontal, we capture the videos from a top view. In order to model the complex fish body deformation, a midline subspace model is firstly learned from a large number of training samples which give a compact representation of fish body and thus facilitate subsequent motion estimation. Then we propose a multicue cost function which is able to characterize the subtitle appearance details of fish body during swimming. Moreover, this cost function is able to work under partial occlusions, making the system fully automatic. Extensive experimental results have demonstrated the effectiveness and robustness of the method.The contribution of the paper can be summarized as follows:(i)
We introduce a midline subspace model learned from large amount of data to model the complex shape and deformation of a fish body. This subspace model is compact and low-dimensional thus greatly facilitating parameter optimization.(ii)
We propose a highly discriminative and robust multicue objective function which models different aspects of the image structures of fish region.(iii)
We have conducted systematic experiments to demonstrate the effectiveness of the proposed method.


## 2. Shape and Kinematic Model

### 2.1. Midline Representation

Since the videos are captured from a top view, the shape of fish on images is approximately symmetrical about its midline as shown in Figure1. So the deformation of the body can be viewed as driven by the midline. We will show later that once the midline is determined, the contour of the whole body shape can be recovered easily according to a reference shape.Figure 1
(a) An example of fish shape. (b) Illustration of the midline representation.
(a)
(b)As shown in Figure1, a midline can be approximated with a chain of n−1 articulated equal-length line segments. Thus a midline is made up of n joints pii=1n: a head point, a tail point, and n−2 middle joints. During one-time-step deformation, the length of each segment is kept fixed. We use Θ=x,y,θ1,θ2,…,θN to denote the parameters of a midline, where x,y is the position of the head point, θ1 is the orientation of the first line segment, and θii>1 is the rotation angle of the ith segment relative to the first one (i.e., the absolute rotation angle is θ1+θi). The first three parameters Θr=x,y,θ1 determine a rigid body transform, and the rest of the parameters Θd=θ2,θ2,…θn account for the nonrigid deformation of body shape. Given the parameter vector Θ, the ith midline points can be recovered as(1)pi=x+∑j=2icosθ1+θj,y+∑j=2isinθ1+θj.The body width of the shape (di) keeps fixed in one time step, so a pair of contour points qi± can be recovered as(2)qi±=pi±di2−sinθ1+θj,cosθ1+θj.Thus with a reference contour and a midline represented byn joints, a set of 2n contour points can be recovered. Given the image at t as It, the tracking problem can be formulated as maximizing probability: pΘt∣It.


### 2.2. Learning Subspace of Midlines

The major defect of the above representation is the high dimensionality, as sufficient number of line segments is essential to guarantee accurate approximation of the fish body shape. In fact, such a representation is redundant as the deformation of the fish is governed by fewer factors. So we seek to embed the deformation parameters into a lower dimensional linear subspace, which can be learned from large amount of training samples.We collectNt=1800 midlines of various postures and perform principal component analysis (PCA) on their nonrigid deformation parameters Θdii=1Nt. We choose k=6 basis Φ=ϕ1,ϕ2,…,ϕk from the PCA results, and thus each Θd is the linear combination of the following basis:(3)Θd=∑i=1kξiϕi=Φξ1,…,ξkT+Θd¯,where the coefficient ξi=Θd−Θd¯Tϕi and Θd¯ is the sample mean. We find that 6 bases are sufficient to approximate a midline to a satisfactory accuracy. In fact, 6 bases account for 99.9% of the variance. Figure 2 shows the training samples of PCA and the first four principal components of PCA. Now Θd is replaced with 6 parameters, and thus the parameters to be estimated can be written as Θ′=x,y,θ1,ξ1,…,ξk. Tracking problem becomes finding the maximum probability of pΘt′∣It.Figure 2
(a) Midline samples for training PCA. (b) First four principal components of PCA.
(a)
(b)


## 2.1. Midline Representation

Since the videos are captured from a top view, the shape of fish on images is approximately symmetrical about its midline as shown in Figure1. So the deformation of the body can be viewed as driven by the midline. We will show later that once the midline is determined, the contour of the whole body shape can be recovered easily according to a reference shape.Figure 1
(a) An example of fish shape. (b) Illustration of the midline representation.
(a)
(b)As shown in Figure1, a midline can be approximated with a chain of n−1 articulated equal-length line segments. Thus a midline is made up of n joints pii=1n: a head point, a tail point, and n−2 middle joints. During one-time-step deformation, the length of each segment is kept fixed. We use Θ=x,y,θ1,θ2,…,θN to denote the parameters of a midline, where x,y is the position of the head point, θ1 is the orientation of the first line segment, and θii>1 is the rotation angle of the ith segment relative to the first one (i.e., the absolute rotation angle is θ1+θi). The first three parameters Θr=x,y,θ1 determine a rigid body transform, and the rest of the parameters Θd=θ2,θ2,…θn account for the nonrigid deformation of body shape. Given the parameter vector Θ, the ith midline points can be recovered as(1)pi=x+∑j=2icosθ1+θj,y+∑j=2isinθ1+θj.The body width of the shape (di) keeps fixed in one time step, so a pair of contour points qi± can be recovered as(2)qi±=pi±di2−sinθ1+θj,cosθ1+θj.Thus with a reference contour and a midline represented byn joints, a set of 2n contour points can be recovered. Given the image at t as It, the tracking problem can be formulated as maximizing probability: pΘt∣It.


## 2.2. Learning Subspace of Midlines

The major defect of the above representation is the high dimensionality, as sufficient number of line segments is essential to guarantee accurate approximation of the fish body shape. In fact, such a representation is redundant as the deformation of the fish is governed by fewer factors. So we seek to embed the deformation parameters into a lower dimensional linear subspace, which can be learned from large amount of training samples.We collectNt=1800 midlines of various postures and perform principal component analysis (PCA) on their nonrigid deformation parameters Θdii=1Nt. We choose k=6 basis Φ=ϕ1,ϕ2,…,ϕk from the PCA results, and thus each Θd is the linear combination of the following basis:(3)Θd=∑i=1kξiϕi=Φξ1,…,ξkT+Θd¯,where the coefficient ξi=Θd−Θd¯Tϕi and Θd¯ is the sample mean. We find that 6 bases are sufficient to approximate a midline to a satisfactory accuracy. In fact, 6 bases account for 99.9% of the variance. Figure 2 shows the training samples of PCA and the first four principal components of PCA. Now Θd is replaced with 6 parameters, and thus the parameters to be estimated can be written as Θ′=x,y,θ1,ξ1,…,ξk. Tracking problem becomes finding the maximum probability of pΘt′∣It.Figure 2
(a) Midline samples for training PCA. (b) First four principal components of PCA.
(a)
(b)


## 3. Multicue Objective Function

To take into consideration the image cues from the whole fish body area as well as some surrounding context, the midlines are extended as straight lines at the head to guarantee sufficient coverage. And each line segmentpipi+1→ is associated with a rectangular region whose width is pipi+1 and length is two times the body width at pi (i.e., 2di). Each rectangle moves in rigid transforms with pipi+1→. Considering the observation that the image likelihood of different parts of the fish body should play different roles in tracking, we divide the points into four parts: Ω=∪i=14Ωi as shown in Figure 3 and sample points are uniformly picked in each rectangle. We compute image likelihood function for each of the four parts, respectively, and then the weighted sum of the functions is computed as the final objective function value. Three kinds of image likelihood which characterize three kinds of information are considered, and they are temporal appearance coherence, segmentation compatibility, and shape self-symmetry.Figure 3
The distribution of sample points and division of the points.


### 3.1. Temporal Appearance Coherence

The appearance coherence is the basic assumption in visual tracking, which enforces the appearance of the estimated target state to be consistent with a reference appearance model. We compute the similarity between the pixel values in each part att and their correspondences at a reference frame t0. The normalized cross-correlation (NCC) is adopted as the similarity metric. For example, the similarity score of the first part is(4)Eapp1=NccI0x∣x∈Ω1t0,Itx∣x∈Ω1t.The scores of the rest three parts can be computed likewise.


### 3.2. Segmentation Compatibility

Segmentation compatibility is introduced to enforce the estimated shape be compatible with the segmentation result. Since segmentation performance is stable across time, enforcing segmentation compatibility will prevent the tracker from drifting. As we select a larger region which contains some context pixels, both the foreground and background should be compatible with the reference. LetBt denote a segmented binary image of It, xit is the ith point in Ω1t, and then the segmentation compatibility score of the ith part can be computed as Eseg_fg1 and Eseg_bg1:(5)Eseg_fgi=i∣Bt0xt0i=Btxti=1i∣Bt0xt0i=1,Eseg_bgi=i∣Bt0xt0i=Btxti=0i∣Bt0xt0i=0,where the first term Eseg_fg forces the estimated shape cover more foreground pixels and the second term Eseg_bg encourages the shape to leave less background pixels uncovered. The segmentation compatibility scores of three other parts can be computed likewise.


### 3.3. Shape Self-Symmetry

So far, we have exploited the appearance coherency and segmentation cues; however, the internal structure of the shape has been ignored. As discussed previously, the shape is self-symmetrical about the midline. So if a midline is correctly estimated, the image structures on the two sides of the midline should be symmetrical. This prior knowledge offers a stable and drifting free guidance for tracking. Like the two previous cues, we compute a self-symmetrical score for each of the four parts. Take the third partΩ3, for example, the self-symmetrical score is computed as the NCC between two pixel value vectors, which are formed by the pixel values of sample points on the two sizes of midline (as shown in Figure 4). The order of the pixel values should be adjusted so that two points that are symmetrical about the midline are in the same position of the array. Formally, the score can be written as(6)Esym3=NccIxi+∣xi+∈Ω3+,Ixi−∣xi−∈Ω3−,where xi+ and xi− are the ith point pair which are symmetrical about the midline. The shape self-symmetry scores for the other parts can be computed likewise.Figure 4
Illustration of the computation of the self-symmetry cue.


### 3.4. Combination of Multiple Cues

The final objective function is the weighted sum of the scores of all the four parts:(7)EΘt′=∑i=14wiEappi+Eseg_fgi+Eseg_bgi+Esymi,where wi is the weight of part i, which is set empirically in the experiments. Different parts should play different roles in tracking, and this has been proven in our experiments. We find that the head (part 1) and tail (part 4) should be associated with larger weights than parts 2 and 3, and this is possibly because the image regions of head and tail contain more discriminative features than the other two parts.


## 3.1. Temporal Appearance Coherence

The appearance coherence is the basic assumption in visual tracking, which enforces the appearance of the estimated target state to be consistent with a reference appearance model. We compute the similarity between the pixel values in each part att and their correspondences at a reference frame t0. The normalized cross-correlation (NCC) is adopted as the similarity metric. For example, the similarity score of the first part is(4)Eapp1=NccI0x∣x∈Ω1t0,Itx∣x∈Ω1t.The scores of the rest three parts can be computed likewise.


## 3.2. Segmentation Compatibility

Segmentation compatibility is introduced to enforce the estimated shape be compatible with the segmentation result. Since segmentation performance is stable across time, enforcing segmentation compatibility will prevent the tracker from drifting. As we select a larger region which contains some context pixels, both the foreground and background should be compatible with the reference. LetBt denote a segmented binary image of It, xit is the ith point in Ω1t, and then the segmentation compatibility score of the ith part can be computed as Eseg_fg1 and Eseg_bg1:(5)Eseg_fgi=i∣Bt0xt0i=Btxti=1i∣Bt0xt0i=1,Eseg_bgi=i∣Bt0xt0i=Btxti=0i∣Bt0xt0i=0,where the first term Eseg_fg forces the estimated shape cover more foreground pixels and the second term Eseg_bg encourages the shape to leave less background pixels uncovered. The segmentation compatibility scores of three other parts can be computed likewise.


## 3.3. Shape Self-Symmetry

So far, we have exploited the appearance coherency and segmentation cues; however, the internal structure of the shape has been ignored. As discussed previously, the shape is self-symmetrical about the midline. So if a midline is correctly estimated, the image structures on the two sides of the midline should be symmetrical. This prior knowledge offers a stable and drifting free guidance for tracking. Like the two previous cues, we compute a self-symmetrical score for each of the four parts. Take the third partΩ3, for example, the self-symmetrical score is computed as the NCC between two pixel value vectors, which are formed by the pixel values of sample points on the two sizes of midline (as shown in Figure 4). The order of the pixel values should be adjusted so that two points that are symmetrical about the midline are in the same position of the array. Formally, the score can be written as(6)Esym3=NccIxi+∣xi+∈Ω3+,Ixi−∣xi−∈Ω3−,where xi+ and xi− are the ith point pair which are symmetrical about the midline. The shape self-symmetry scores for the other parts can be computed likewise.Figure 4
Illustration of the computation of the self-symmetry cue.


## 3.4. Combination of Multiple Cues

The final objective function is the weighted sum of the scores of all the four parts:(7)EΘt′=∑i=14wiEappi+Eseg_fgi+Eseg_bgi+Esymi,where wi is the weight of part i, which is set empirically in the experiments. Different parts should play different roles in tracking, and this has been proven in our experiments. We find that the head (part 1) and tail (part 4) should be associated with larger weights than parts 2 and 3, and this is possibly because the image regions of head and tail contain more discriminative features than the other two parts.


## 4. Sequential Particle Swarm Optimization

With the objective function defined, the tracking problem becomes maximizingEΘt with respect to the parameters Θt′. However, this objective function is highly complex and nondifferentiable. Particle Swarm Optimization is a stochastic optimization technique which has received much attention due to its ability in finding optimum of complex problems.We adopt a standard particle swarm optimization procedure [26]. A set of candidate solutions are maintained as particles, which move in the parameter space in the influence of a global best solution and a particle’s local best solution. In each time step, the tracker generates initial solutions using a second-order dynamic model:(8)Θt′=2Θt−1′−Θt−2′+nt−1,where nt−1∼N0,Σ is the Gaussian noise.


## 5. Experiments

We captured the data for evaluation: 20 zebra fish were placed in a water tank and a video camera was placed on top of the tank which recorded the movements of the fish. The resolution of the camera is 1024 by 1024, and the frame rate is 100 fps.We first evaluate the method following the manner of conventional multiple target tracking. The tracking of a target is considered to be completed if no ID switch or target lost occurs during the entire time step. We have manually labeled a 650-frame long video for automatic evaluation. A tracker is considered to be failed if both estimated shape’s head and tail positions are more than 20 pixels away from the labeled positions which are considered as groundtruth. We get an 800-frame video with 10 zebra fish, which is shot in front light. The resolution of the video is 2048 W by 2040 H, and the frame rate is 100 fps. On this data set, we evaluate the adaptability of our method.In order to evaluate the role of different cues, we evaluate different combinations of cues on the labeled data. The evaluation results are listed in Tables1 and 2.Table 1
Evaluation results of different cues on video of 650 frames.
Methods
cue1 + cue2
cue1 + cue3
Full
IDS
4
6
2
Lost
2
2
1
Completed
14
12
17
Total trajectories
20
20
20Table 2
Evaluation results of different cues on video of 800 frames.
Methods
cue1 + cue2
cue1 + cue3
Full
IDS
2
2
1
Lost
1
1
0
Completed
7
7
9
Total trajectories
10
10
10We evaluate different methods on the labeled data and the 800-frame video. The evaluation results of different methods are listed in Tables3 and 4. Qian’s method is more prone to IDS and lost when the number of frames occluded by each other increase. In Wang’s method, the lower image resolution possibly leads to the worse effect of CNN (1024 ∗ 1024 vs 2048 ∗ 2040). The proposed method utilizes more body shape information and thus has better performance.Table 3
Evaluation results of different methods on video of 650 frames.
Methods
IDS
Lost
Completed
Proposed
2
1
17
Qian et al. [23]
3
3
14
Wang et al. [25]
3
2
15Table 4
Evaluation results of different methods on video of 800 frames.
Methods
IDS
Lost
Completed
Proposed
1
0
9
Qian et al. [23]
2
1
7
Wang et al. [25]
1
1
8To evaluate the accuracy of the estimated shape, we give a plot of errors of the head and tail point of one fish in Figure5. From the figure, we see that the error does not accumulate as time grows. The estimated tail point vibrates more violently than the head point, and this is possibly because the appearance of the tail is not quite stable as that of the head point.Figure 5
Error plot of the tracked head and tail positions of one target by our method.We also give some qualitative results of the tracked shape under complex mutual interactions in Figure6. With the designed multicue objective function and midline-based subspace model, our method is able to overcome medium degree of partial occlusions. And we find our method may fail if the occluded area is too large. Finally, we plot the trajectories of the estimated head position of 20 fish in Figure 7. For better visualization, we added time as the third dimension of the plot.Figure 6
The qualitative results of our method when fish has complex interactions. (a) #150. (b) #160. (c) #170. (d) #180. (e) #190. (f) #200.
(a)
(b)
(c)
(d)
(e)
(f)Figure 7
The resultant trajectories of the fish head by the proposed method.


## 6. Conclusion

We present in this paper a method that is capable of tracking the full-body deformation of multiple fish with frequent occlusions and interactions. We propose a midline subspace-based model to represent each complex shape and deformation of the fish. And we further propose a PSO-based multicue optimization method to estimate the parameters of the model. Experimental results demonstrate the effectiveness of the proposed method.



---

*Source: 1868797-2019-06-24.xml*