# An Improved Optical Flow Algorithm Based on Mask-R-CNN and K-Means for Velocity Calculation
## 2 Algorithm Design
### Pyramid LK Algorithm  
The optical flow algorithm can be used to calculate the instantaneous velocity by comparing the corresponding features in two adjacent images. It estimates the optical flow value based on the assumption that the brightness of the feature pixel points does not change [32] between two adjacent images. The instantaneous velocity is called the optical flow vector. When the sampling interval \(\Delta t\) between two adjacent images tends to 0, it can be considered that the gray level of corresponding points in the two images does not change, which can be expressed as:  
\[E(x+u\Delta t,y+v\Delta t,t+\Delta t)=E(x,y,t) \tag{1}\]  
where \(x\) and \(y\) represent the horizontal and vertical positions of pixel points respectively, and \(E(x,y)\) represent the brightness of point (\(x\),\(y\)) at time \(t\). The brightness along the trajectory does not change. The equation can be expanded as:  
\[\frac{dE(x,y,t)}{dt}=0 \tag{2}\]  
The differential form of Equation (2) is:  
\[\frac{\partial Edx}{\partial x\partial t}+\frac{\partial Edy}{\partial y \partial t}+\frac{\partial E}{\partial t}=0 \tag{3}\]  
When the gray value of the image changes with the three variables of \(x\), \(y\), \(t\), the left side of the Equation (1) can be expanded by Taylor series:  
\[E_{x}u+E_{y}v=-E_{t} \tag{4}\]  
where \(\frac{\partial E}{\partial x}=E_{x}\), \(\frac{\partial E}{\partial y}=E_{y}\) denote the horizontal and vertical gradient of pixel; \(\frac{\partial E}{\partial t}=E_{t}\) is the time gradient of a pixel; \(\frac{dx}{dt}\) and \(\frac{dy}{dt}\) are the horizontal and vertical components of the velocity vector, which is called optical flow. Due to the negligible time, motions between two frames can be regarded as linear in a relatively short time, thus \(\frac{dx}{dt}=u\), \(\frac{dy}{dt}=v\). If the equation is set up with N identical pixels, the following can be obtained:  
\[\left[ \begin{array}{cc}E_{x_{1}}&E_{y_{1}}\\ E_{x_{2}}&E_{y_{2}}\\ \vdots&\vdots\\ E_{x_{3}}&E_{y_{3}}\end{array} \right]\left[ \begin{array}{c}\mathbf{u}\\ \mathbf{v}\end{array} \right]=\left[ \begin{array}{c}E_{t_{1}}\\ E_{t_{2}}\\ \vdots\\ E_{t_{3}}\end{array} \right] \tag{5}\]  
where the solution of equation that means the moving velocity \(\mathbf{u}\) and \(\mathbf{v}\) is easy to calculate. Figure 1: The process of improved algorithm based on Mask-R-CNN and K-Means. The precision of LK algorithm is not adequate when processing two adjacent images. To solve this problem, the Gaussian image pyramid technique is introduced, which satisfies the small motion hypothesis by continuously subdividing the velocity and improves the noise immunity of the algorithm. The core idea of the pyramid LK algorithm is to construct the image to be compared into an image sequence with a resolution ranging from small to large. Firstly, the image with a low resolution is calculated, and then the result is iterated as the initial value to the image with a high spatial resolution. That is, from the top layer of the pyramid to the bottom layer until the lowest value of optical flow value is calculated. The process of pyramid LK algorithm is displayed in Figure 2.  
**Step 1**: for the adjacent reference frame and the current frame in the image sequence, the levels of different resolutions are sampled according to the decreasing precision, and the original image is taken as the base of the pyramid (\(L=0\)). When the image is divided into a certain level, the motion parameters with large displacement will become small enough to meet the constraints of optical flow calculation. **Step 2**: calculation, and in turn down from the top layer set \(g_{L}\) as \(L\) initial estimate optical flow vector, \(\Delta f_{L}\) as optical flow calculated in \(L\) layer as a result, the mapping relationship of between the layers is:  
\[g_{L-1}=k(g_{L}+\Delta f_{L}) \tag{6}\]  
**Step 3**: set the total number of pyramid layers as N and initialize the optical flux \(g_{N-1}\)=0 at the top of the pyramid.