# IOF-Tracker: A Two-Stage Multiple Targets Tracking Method Using Spatial-Temporal Fusion Algorithm
## 2 Methods
### Fusion Model of R-IoU and Optical Flow Features  
#### 2.1.1 Optical Flow Features  
Optical flow [14] is caused by the movement of foreground objects, camera motion, or both. The optical flow feature is represented by assigning a velocity vector to each pixel in the image. These velocity vectors form the optical flow field. In the absence of moving objects, the optical flow field usually appears continuous and uniform. However, when moving objects are present in the image, their optical flow features will differ from the stationary background, resulting in discontinuous and uneven characteristics in the optical flow field.  
In this section, we optimize the optical flow field calculation based on [PERSON]'s optical flow algorithm [15]. [PERSON]'s algorithm estimates optical flow by polynomial fitting for each pixel and its neighboring pixels. However, due to the large number and complexity of pixels in the target area, the optical flow vectors obtained for each pixel are significantly affected by noise, making optical flow feature matching difficult. Therefore, we optimize this by averaging the optical flow vectors obtained for each pixel, generating a mean optical flow vector with less noise.  
Since the [PERSON] optical flow method is already integrated into the current OpenCV, we only provide the basic steps for calculating the optical flow vector of a moving target as follows:  
Assume that the image sequence is denoted as \(I(x,y,t)\), where \(X=[x,y]\). An image sequence is represented by each consecutive frame extracted from the video. Assuming constant image brightness, i.e., there is no change in image brightness, the derivative is 0. The formula is as follows:  
\[\frac{dI(X,t)}{dt}=\frac{\partial I}{\partial X}\frac{\partial X}{\partial t}+ \frac{\partial I}{\partial t}=0 \tag{1}\]  
where, \(\frac{\partial X}{\partial t}\) represents velocity in a small time interval, which can be denoted as \(\frac{\partial X}{\partial t}=\left[\frac{\partial x}{\partial t},\frac{ \partial y}{\partial t}\right]=[\vec{u},\vec{v}]\). Then, the following equations can be derived:  
\[I_{x}u+I_{y}v+I_{t}=0 \tag{2}\]  
\[\left[\begin{array}{cc}I_{x}&I_{y}\\ \end{array}\right]\left[\begin{array}{c}\vec{u}\\ \vec{v}\\ \end{array}\right]=-I_{t} \tag{3}\]  
By following the basic steps for solving optical flow as described above and combining them with the Farneback optical flow method, we can obtain the horizontal component and vertical component of the optical flow for each pixel.  
For a certain pixel \((k,l)\), its optical flow vector can be expressed as \((\vec{u}_{k,l},\vec{v}_{k,l})\), where \(\vec{u}\) and \(\vec{v}\) represent the motion components of the pixel in the horizontal and vertical directions, respectively.  
Since the magnitude of the resultant vector can be directly used to detect the intensity of motion, such as determining whether there is noticeable motion in a certain area. Direction information helps analyze the movement trend of objects: For example, whether the object is moving left, right, up, or down. The direction information better describes and matches different motion patterns. Therefore, we process the \((\vec{u}_{k,l},\vec{v}_{k,l})\) components of each pixel point as follows.  
First, we perform the average optical flow vector as shown in Equation (4).  
\[\vec{u}_{\text{avg}}\,=\frac{1}{P\times Q}\sum_{k=1}^{P}\sum_{l=1}^{Q}\vec{u} _{k,l}\quad,\quad\vec{v}_{\text{avg}}\,=\frac{1}{P\times Q}\sum_{k=1}^{P} \sum_{l=1}^{Q}\vec{v}_{k,l} \tag{4}\]  
where \(P\times Q\) is the number of pixel points in the target area.  
Then, the average sum vector \(\vec{M}_{avg}\) is expressed as depicted in Equation (5).  
\[\vec{M}_{avg}=\left(\vec{u}_{avg},\vec{v}_{avg}\right) \tag{5}\]  
The final achieved effect is shown in Figure 2.